Best Practices IBM Adopts Within Their Open Source Program Office

Best Practices IBM Adopts Within Their Open Source Program Office

July 1, 2021
9:00 am

About

Today I talked with Javier Perez, the former Director of Product Management at RedHat.

In this podcast, we dive deep into what his current role entails as an Open Source Program Strategist at IBM and discuss best practices IBM adopts within their open source program office. Javier is an expert when it comes to open source security, so it was only natural that we went into detail about why so many security vulnerabilities exist within open source and what developers should do about it.

Transcript

hey everybody welcome back to the 22nd

episode of open source for business

brought to you by open teams my name is

henry badri

and today i talked with javier perez the

former director of product management at

red hat in this podcast we dive deep

into what his current role entails as an

open source program strategist at ibm

and discuss best practices ibm adopts

within their open source program office

javier is an expert when it comes to

open source software security

so it was only natural that we went into

detail about why so many security

vulnerabilities

exist within open source software and

what developers should do about it

whether you

are a user developer manager or just

curious about the industry

open teams is the place to find the

information news

training and support you need to thrive

with open source software

now the introductions are out of the way

let’s dive in

javier thanks for so much for joining me

on the podcast today hey henry

got to talk to you finally uh had a

chance to

have a conversation with you yeah i’ve

been super excited to chat to you

i know that you’ve led product teams at

over five different companies uh one of

them being red hat where

you were the director of product

management for more than two years

so can you talk about how your journey

went into open source

as a product focused leader so that we

can get get a better idea of how you

actually got here today

sure you know i started working for for

startups and

and i worked for a couple of startups uh

one of them

the name was feet henry i can tell you

the story about the name

later um and we built uh

an offering we built a cloud-based

offering a product

a platform uh 100 based on open source

software 100 based on open source

software

and uh that led us you know very proud

of that that

offering the group effort as a team

uh that led us into an acquisition by

red hat so that’s how i i went into into

red hat and

i went there and and obviously much more

focus on open source and i had the

opportunity to learn a lot about uh

you know more of the insight and how to

work with some of the communities

and how to contribute and many many of

the the

different things that red hat those with

the different communities

had a chance to to experience that uh

from product management perspective you

have to wear the two hats and it’s

always the

interesting piece right you have to keep

promoting the

the the open source adding more open

source software

and at the same time you have to find a

good way to monetize that open source

and and go from an

open source project into a product right

a commercial offering that’s really

interesting and so

what does that phase look like or what

did that look like at feed henry and a

red hat

yeah we we had um you know one of the

models and open source

is to go to the cloud right so you offer

the software

uh it’s out there it’s open uh anyone

can contribute and

hopefully you start building a good

community of uh

users and downstream users but at the

same time also contributors upstream

contributors

and uh and but you have to have the

balance right of obviously you you

if you are working on as part of a

business and and it’s part of your model

working with open source and having a

commercial offering

we went through the route of our

cloud-based or the hosting which is

you know it’s it’s a successful model

and many open source projects have gone

that way right

the software is open source uh but we

charge you a fee for the hosting you

don’t have to

worry about configuring setting it up

maintenance we

we take care of that and so we went in

with that route

um then at red hat um

we continued that and it was very

interesting on the red hat side because

you know we also had other products

right that open source product or

projects

that that we wanted to complement from

from red hat’s point of view so we went

through a

good exercise of moving everything into

containers and to obviously uh openshift

before that we we our engineering team

did a fantastic job

working with uh with containers for

example when

containers were not

that known right actually even before

docker containers and before

kubernetes orchestration so we were

doing some of that work some of that

inspired by

you know the likes of heroku some of you

might remember heroku

and uh now out of salesforce um

so we were doing a lot of all that the

same same goes for node.js

we were uh working with node.js at on

the early days

and and i remember checking the npm

you know packages and the npm registry

or how many new packages we have this

month how many new packages

i will go and tell our customers hey you

see now we have

you know 10 000 more than last time we

spoke about and then you know i lost

count but obviously now we are in the

millions i believe it’s more than 1.2

million

packages on npm which is just amazing

growth and you know overall open source

but uh npm specifically for

for javascript and node.js that would

have been really exciting just to be at

the

leading edge of that technological

revolution because everyone’s using

containers today it’s just it’s kind of

become the industry standard

and when you were at red hat

how did you see the company evolve

through to

today because i know that you joined in

2014 yeah

uh you know i haven’t been there for for

for a

few years um what is it what four or

five years

uh but now i’m with ibm which you know

red hat then became part of ibm so i’m

actually working very closely with some

of the the initiatives there

between the two organizations uh one of

the things that i

noticed obviously is is the the growth

of openshift

um you know when i was there four or

five years ago

uh we we all knew that that was the next

big thing we all knew that

you know everyone was moving into

containers and

as you one thing is to to you know

build your app host your app on one

container or maybe a couple of

containers

that’s one thing but for very large uh

uh workloads for very large applications

uh you needed a lot more than that right

you needed a full orchestration and you

need to build

more kind of enterprise great features

on top of what you have today and

that was the the premise of uh openshift

right

make kubernetes more enterprise grade we

add more

security features we have more uh

you know overall ecosystem around around

that so

that has that has been the difference

right linux obviously

red hat support for linux and

contributions to the linux kernel and

and linux subsystems that continues

um red herring spread linux obviously

that will continue to do well

uh pretty much everything is linux

nowadays

and uh and but the big difference for me

has been

uh how much openshift has grown

uh in the last five years uh which is

just amazing and

since the ibm acquisition uh well now

they have access to all those ibm uh

clients uh customers all over the world

right so uh

even the strategy the ibm study it’s all

about

um you know hybrid cloud which is all

based on the same

what i call the building blocks right

the operating system

the container the orchestration of

containers the

you know the the open source

uh storage solutions and

integration solutions and runtimes and

you know what we just call middleware

all of that all those all that is stack

it’s

uh you know as well said and it’s all

based on open source so so that those

are going some of the major differences

obviously you know at that much larger

scale nowadays

what’s your favorite of the two colors

red or blue

what should i say purple now purple yeah

i think that’s i think that’s a good one

you’ve shifted from from red to blue and

i guess now i’d like to shift gears and

focus on your time

sitting in the area of the blue so i

know that you’re currently

an open source program strategist at ibm

so what’s that role entail yeah it’s a

it’s a very exciting very interesting

role

after what is it 10 12 years running

product management

uh product management teams responsible

leading projects and products

uh went into this role which was very

very exciting

to to me and it’s about promoting open

source

uh ibm being such a large organization

i’m part of the

the the ibm z organization or that

organization

um and it’s all about promoting open

source uh ibm’s been in open source for

many many years more

than 20 years actually about 20 21 years

just on linux

but wait we can track open source way

back to the 1950s actually i have a

really good presentation when i talk

about

okay brief history of open source and

and it all started with the mainframes

back in

1955 right and and by the way there’s an

organization called

share which started back then and

it’s still around and doing really great

stuff

uh all about promoting uh uh sharing

software and actually that’s by the name

they said

it’s a it’s it’s what we do it’s not an

acronym it’s

what we do we share uh software and and

ideas

so my role it’s it’s internally is to

promote open source

open source internally uh there’s as you

can imagine different

levels of maturity within the the teams

uh some of them very new to open source

some of them

experts contributors uh maintainers of

some of the projects right

so you i work with with them at the

different levels uh and

externally i do a lot of advocacy uh i’m

doing now now that we’ve been still

in this pandemic i’ve been doing

webinars you know all the time

i supposed to be traveling to different

events but the webinars

blog post uh interviews podcast all of

this

uh which i enjoyed very much and it’s

all about talk

all about promoting the work that it’s

uh i mean or promoting open source and

also the work that

we do at an ibm uh promoting open source

including open source for the different

platforms hardware platforms

uh platforms like you know the ibm z and

the linux one

and the uh power and others and i

definitely know ibm

has been at the forefront of this open

source revolution ever since

i think it was 2000’s or maybe before

they they have always taken it very

seriously and sort of been a leader in

that front

i was gonna tell you the just just that

you mentioned that uh quick story with

linux for example

um back in in uh 2000 1999

1999 was the first time that linux was

installed was deployed in a mainframe

okay mainframe you know known as the

property operating systems property

hardware

linux was installed there for the first

time and then in 2000

uh ibm uh that was very early on for for

linux

ibn made the decision to invest on linux

uh across all platforms right across all

hardware platforms

uh they invested a billion dollars

on linux back in 2000 wow um

with that money uh you can buy uh super

commercials actually there’s there’s one

really really good

that the linux foundations like to show

from time to time

you can buy super bowl commercials but

you can also do a lot of investment

on the different platforms right and you

can hire

engineers you can hire developers to

work upstream to work full-time

on uh on open source projects including

the linux kernel including some other

subsystems within

the linux so i i found it that when i

when i found when i heard about this

story was like

that’s we have to keep telling this

story this is this is great

right and uh and uh you know

the benefits of that it’s across the

world right so for every

developer that is contributing that is

sending a pull request with an

enhancement or a bug fix or a

vulnerability fix

that benefits everybody else including

you know all the different linux

distributions that are out there

right so it benefits uh

ibm it benefits the platforms of ibm

it’s it’s offering but it

benefits everybody else as well so ibm

definitely took that leap at the

beginning before anyone really saw

the value of open source because for

years i think at microsoft at that time

was still saying open source was a

cancer

or just before that they were so they

were they were still shocked shocked by

it but it’s very interesting to see that

ibm has had

three or two decades of of running an

open source program office when did

their open source program office start

do you know

i don’t know exact date but i i can tell

you that it’s a really

formal uh well-organized

office uh that was one of the for me

joining ibm that was

one of the the good

news or the good surprises uh for me

um there there’s it’s it’s well

organized

uh for such a large organization it’s

really uh driving uh open source and i

can tell you a little bit of a few more

things around

you know what what what we do as part of

the open source program office

definitely i’d love to also know some

best practices because i feel like um

people listening could definitely

benefit from that yeah i mean best

practices

and and you know maybe some of the

things that that we’re doing

at ibm that maybe some some other

organizations are not doing right

i mean of course you have to have

certain uh governance

and uh you don’t want i mean this is not

unique to ibm this is i’ve seen it on

many organizations including red hat and

many others

where you know there’s a level of

government governance right

uh you encourage your your employees

to go and work on open source work on

whatever project you want

uh you know work if you want to work

after hours or

on weekends you know that’s that’s good

too right you

you don’t need to tell that you’re

working for ibm or use your ibm email or

anything like that we encourage

our employees especially developers

engineers to go ahead and

work on on the projects that they are

interested in

why would you do that then someone will

ask why would you do that right

and and that’s motivational that’s

that’s i’ve seen it for years right

engineers developers that that are

working on something that that is of

their interest that they feel like

they’re learning

and they’re feeling that they’re doing

something uh

good positive innovative uh they

they keep they kept themselves motivated

right they’re not gonna be working from

nine to five

uh just a regular job they’re gonna be

working more hours they’re gonna work in

overnight they’re gonna work

on weekends because that’s their their

becomes a passion right so

uh encouraging that culture it’s it’s

it’s been great

um we have just just to mention a couple

of things we

we have what we call uh open source

dojos

dojos for basically bringing uh the

master or the expert

uh some of the most experienced uh kind

of open source contributors

they go on and basically volunteer for a

couple of hours a week or maybe an hour

a week

uh typically it’s done over slack slack

channel and

anyone can you know sign in on those

hours and

ask any questions right you know i have

trouble with with this

uh this community or i you know

my pull request was rejected what should

i do here or that or

you know i never i haven’t come across

this open source license or anyone has

experience with this license what should

i do on this

so the type of questions that anyone

will have

we assign this you know experts the

people that have the most experience

to lead these uh kind of what we call

open source uh dojos and of course we

have

uh some of the experts more on the let’s

say on our ai

related open source or some of the more

on the devops related open source and

some other more on the

you know integrations type of uh

opposite languages programming languages

experts

uh so that’s that’s a fantastic practice

i i think uh it is that’s going really

really really well um

the the other uh interesting piece and

and

is is training um actually ibm made

mandatory open source training uh you

have to do it once a year every year so

you have to renew your your training

and you cover all the basics of open

source which is

it’s good refresher for for anyone all

the basics you know

from from you know what exactly what

exactly we mean with open source

software to

you know basics of the open source

licenses to

you know being a good open source uh

neighbor a good

uh citizen right where it’s not just

download and

work downstream but also go back and

contribute and you know

making very clear to people that hey if

you fork something

now you’re on your own right now you

have to maintain that fork now you it’s

it can become closed source now

so we kind of refresh on that uh every

year

and that training is taken by by

thousands of

uh ibm employees which is fantastic it’s

all online

it takes uh it takes uh i think it’s

it’s a

half an hour an hour results to do it

and you do it once a year

how has that training impacted the

organization internally it seems that

like that is kind of an inner source

approach

yeah so inner source it’s it’s going

really well by the way there are over

also inner source initiatives within

different groups

uh i i recently uh heard a presentation

from a team that

did everything on a inner source model

right which for people that are not

familiar it’s it’s it’s exactly the same

as open source the only difference is

that is within the company right

you try to bring contributors you your

code is

available to anyone the only difference

is inner sources within your your own

company

but the other thing is that we also try

to promote that you know okay you are

done with your inner source project or

you know

release now let’s open source that right

because we see all the other benefits of

building that community and bringing

more people to

to to work with with that uh in terms of

the benefits for

for training it’s you know it’s one of

those things that it’s always hard to

to quantify right uh you know it’s like

asking for return of investment uh

on these things it’s it’s always kind of

difficult to uh

to to quantify but i’ll tell you that

um and i i’m i’m always saying there’s

never enough good communication right

you have to go and tell them or train

them enable people

again and again and again right uh

because you know everyone has

like their day jobs and uh and uh

and and you know if you’re not doing it

do you forget right

so uh it’s good to keep having it uh

it’s good that uh your employees know

what when you talk about open source and

when you talk about

as i was mentioning earlier doing a blog

post or an announcement or a press

release

about a new offering that comes with all

the innovation from open source

so it’s good to have the rest of your uh

company and your employees to

to talk about that as a good thing right

uh all right we are innovating uh we are

promoting innovation and and by the way

another interesting thing about ibm is

we also have a very very strong research

team

that works on many patents right

actually i believe

ibm is the number one company in the

world in terms of the number of patents

per year

and and we are top one top two top three

in terms of open source contributions or

contributors to open source

as well so so how come these two which

seem to be completely opposite right

closers

you have a patent versus open source and

you know the

the same research thing that it’s uh

that has all many patents and continue

to work on patents

that same research team also works on

open source because they see the benefit

of you know growing the community

bringing more contributors

so um it is a nice balance that

companies

have but nowadays right and we have so

many more recent examples where

companies try to kind of be more

restrictive on their licenses

or try to well we can’t just be

completely open source well

you have to have a balance that’s humble

that’s that’s that’s understandable

um you want to monetize uh your the work

that you’re doing

um but having your employee base having

your

engineers all familiar with open source

i think that’s that promotes innovation

and and that’s that’s been uh that’s

something i know an old thing that i’ve

seen

like a really really positive for for an

organization and especially a large

organization like ibm that’s really

interesting i know when i first heard

about the defensive pattern strategy of

ibm

i didn’t know it was largely defensive i

thought they were taking out patterns

which

i believe largely hinders hinders in

innovation a lot of

a lot of senses but i heard that it was

defensive to help the open source

community which i thought was

a great initiative and i know that one

other topic that you have quite a bit of

experience in is security

open source software security to be

specific how do you

identify vulnerabilities in open source

software and

why do these vulnerabilities and

exploits exist

good good point and uh i am glad i have

the opportunity to talk about this i

i every time i have a chance i you know

do a

or presentations about open source

security and some people will say well

everything is open source right so i’m

writing i’m coding on

um java i’m coding on ojs and coding on

python and

and i’m dependent i depend on all these

libraries right i can’t just write

everything from scratch i have to use

these

these libraries and depending on the

programming language

you have you can have a few and you can

have hundreds

or even thousands of open source

libraries right and then the beauty of

open source is

since you have so many people working on

on on and contributing

the beauty is that that you know one

open source project

uses other open source project which

uses another open source project

and then you have a chain right of all

these dependencies and that’s what we

call you know

all the library dependencies or

transitive libraries

so with all that grow if you have a

vulnerability in one of those

dependencies it affects everything else

right it could affect

thousands uh i actually have a really

nice image of that looks like a star but

it just

in a graphic way shows all the

dependencies on on

on the libraries and the dependencies on

between between libraries

so first uh why it

it becomes out of control it’s not our

controller but it’s just

you are you are reusing open source from

everybody else

right that’s just the nature of the

nature of the beast and the same happens

with close source by the way

because you also create libraries and

for the most part

on any or most closed source software is

using open source library especially the

ones that they relate to

to your programming languages right and

so what happens there’s not a especially

for the large

larger projects right there’s not um a

single architecture

right have many contributors all

building right so as supposed to have a

single architecture and everyone moves

in that direction

you can have contributors building you

know one subsystem one another

functionality one enhancement

so the fact that you have many

contributions and and you start

growing your open source software your

project uh then that means that

that you might get you may have some

gaps there right as not as a single

architecture

those gaps can become vulnerabilities

the other important point is is really

the the lack of

um kind of security knowledge right

talking about training and talking about

enablement developers

there are some basic uh kind of security

um uh items that the developers

should learn for example you can start

with the top 10

owasp vulnerabilities right

if they if the developers know about

that

then they’re going to pay attention to

those potential vulnerabilities and

they’re going to write a better code so

so training uh uh it’s it’s key

the the only problem by the way some

companies will say well like

impossible to bring you know my entire

teams for training and things like that

uh a lot of companies are doing the the

concept of uh uh security champions

right so you have

one person on each team on each scrum

team or any other team

and that’s your your security expert

that’s the one that is going to go and

review or answer questions around

well let’s make sure that you don’t have

a uh

scripting uh cross-site scripting

vulnerability here or making sure that

you don’t have a

sql injection or any type of injection

into your code

and and they’re things that are kind of

relatively easy to identify with some

with just a little bit of training so

the fact that the architecture

is not one piece it’s put it together in

multiple pieces

the fact that there’s there’s some lack

of a kind of security basics

training that makes possible that they

are google abilities

all the time now or you can find

vulnerabilities

all the time now the smart thing to do

and and you know many developers know it

and do it

is i discover a vulnerability i don’t go

and tell you that there’s a

vulnerability i go and work on the fix

once i have the fix then i go and

disclose the vulnerability with here’s

the fix

which is great that’s the proper way to

do it actually the proper way to do it

will be also to disclose

the vulnerability to the national

vulnerability database or some other

repository so so other people know about

the

not just on you know alignment your code

or a comment in in your code

uh and then the last piece just to uh

complete here my

my comment it’s uh not every

vulnerability

it’s ex it’s been exploded right so one

thing is

you know you the analogy that i like to

say is you leave the door

wide open in your house or your

apartment

that that’s a vulnerability right that

doesn’t that doesn’t mean that something

bad is going to happen

right it’s just some vulnerability you

left the door wide open

now if someone comes in and steals

something that’s an exploit on your

vulnerability

right so the same happens in uh with

code uh

you can have you know a number of

vulnerabilities in your code

uh that doesn’t mean the salary that

you’re gonna explore that that you’re

gonna have explored like um and the

other thing is

exploits could happen in many different

ways which that’s another big problem

that we have in open source and then

open source

security which is you know we can be

take care of uh vulnerabilities but

maybe there’s a different type of

exploit that we didn’t

that we didn’t know right so so it gets

really really complex

and it’s getting very very interesting

and by the way that’s another reason why

the

cyber security especially the

application security space it’s

it’s hot it’s it’s growing and also in

the open source side of things

that’s great to hear and i’m a bit

worried to see what happens in the

future i think that with all this open

source and just our

entire lives built upon software there

are definitely going to be many

opportunities to

exploit the vulnerabilities so i hope we

stay ahead of the hackers or the people

that are going to exploit it

but yeah thank you so much for your time

javier it’s been great chatting with you

absolute pleasure and i hope we get to

do this sometime soon again

happy to do it henry thanks for the

invite thank you and to everyone

listening

uh if you like this then please leave a

like on youtube if you’re watching it

there or if you’re listening to an apple

podcast then please

leave us review letting us know what you

think it really does help out to get the

podcast out there

so thanks everyone thanks javier and

until next time

Resources

Company