About
In this episode, we were joined by Pablo Fuentes and Brendan Collins, who told us about the future of the Xarray-Spatial project. Xarray-Spatial implements common raster analysis functions using Numba and provides an easy-to-install, easy-to-extend codebase for raster analysis.
xarray-spatial grew out of the Datashader project, which provides fast rasterization of vector data (points, lines, polygons, meshes, and rasters) for use with xarray-spatial.
xarray-spatial does not depend on GDAL / GEOS, which makes it fully extensible in Python but does limit the breadth of operations that can be covered. xarray-spatial is meant to include the core raster-analysis functions needed for GIS developers / analysts, implemented independently of the non-Python geo stack.
Do you use open source software? Find trust open source experts for any project or question today: https://openteams.com/
Go to OpenTeams (www.openteams.com) to find your Open Source Architect to train, support, and deliver your software solution. Assemble the right open source expert team today.
Thanks for listening!
Transcript
hello the internet welcome to open source directions hosted by open teams the first business to business
marketplace for open source services open source directions is the webinar that brings you all of the news about
the future of your favorite open source projects my name is melissa i’m a mathematician and software engineer
based in brazil working with open source software and communities at quansite and
i’ll be your host for open source directions co-hosting with me today is
hi i’m a deacon i’m excited to be the co-host for this episode of open source directions i’m a postdoc at the
university of illinois at urbana-champaign i work on an open source project called the yt project and i’m based in urbana
illinois hi i’m brennan collins i’m the co-founder and principal at make path
and also the leader of an open source project called x-ray spatial and i’m uh so thankful to be here thank you guys
for uh for inviting me on and really happy to be here to evangelize x-ray spatial
glad to have you um but before we get into the details about x-ray spatial we
are going to do our pr of the week section where we each discuss like a fun little pr that we saw
during the week so brandon you want to get first um sure i don’t mean to focus too much on
x-ray spatial too early but the pr that i wanted to highlight is within x-ray spatial um it’s a pr
um submitted by uh um by tweed.m who’s uh
a very active developer in x-ray spatial which adds gpu support for our hill
shade function so the hillshade function will take an x-ray object or a numpy array and produce uh one that looks
illuminated by a light source and so twee has taken that and enabled that for
cuda gpus so that’s an exciting pr at x-ray spatial
that’s really cool so i just want to mention a quick one which is uh
i’m actually cheating because it’s a pr but i’m it’s a pr for a blog post which
i found pretty awesome about designers and open source uh so about the roles that designers
play in open source communities and how can they better communicate with developers and how do they find out
their own place in open source communities um this is a blog post written by tim george and isabella
precedo floyd from consider and i just thought it was awesome and just
wanted to share medicine do you have something well uh i’m going to say that there’s so
many great uh pull requests out there that i couldn’t possibly choose between all of them slash uh
i was very indecisive um so i would just like to highlight an open source uh news of the week which is
uh the new cci funding for numpy so i know melissa is going to be working on
that so congratulations numpy yeah yeah great everybody
yes we need funding for open source so uh congratulations numpai i’m very proud
of you that’s great and to yes and that one um also congratulations to bokeh on the czi
uh grants um bokeh also received a grant um which which make path will be participating uh on in helping to add
better support for publication ready out exports from bokeh plots oh excellent
yes i think uh the publication ready plots is like super important in database you know otherwise then you can
like actually communicate science so that’s fantastic congratulations also okay and by extension uh
make path yeah thanks for the bouquet team for that yeah i think there’s a congratulations
to all the grantees there were a bunch of really nice projects like and also the ones that maybe
weren’t selected but who definitely deserved uh and also deserve funding so let’s try
and make that happen for everybody would be nice to have more funding for open source projects absolutely
um okay so i think it’s time we get to xray spatial um
this is a raster-based spatial analysis library in python it implements common
raster analysis functions using numba and provides an easy to install easy to extend code base for raster analysis
it has about 150 stars on github and
we are counting 160 downloads a month across pipei and conda um i’m pretty
sure this is uh like used for research for industry for a
bunch of other applications um so we’re very happy to have you brandon to talk about it
oh thank you yeah no we’re super excited brendan um can you tell us a little bit about who
started it why it was started and what needed bills yeah sure so um x-ray spatial is a suite
of raster analysis tools the history of the library has a lot to do with the data shader project so data
shader is a general purpose rasterization pipeline written in python and that’s uh sponsored by anaconda i
was you know founded by peter wang and jim bednar i had the privilege of working a lot on
data shader in the early days and as we went i love geospatial technology and i
always wanted to add in new geo features into data shader and it became apparent that the mission of data shader was
really about a rasterization pipeline not about spatial analysis functions so
as we grew a suite of these analysis functions we then moved those into a separate library that i started and
maintained and jim bednar was really helpful and anaconda was very helpful in getting that project launched
and being able to start doing things like deprecating those analysis functions within data shader and moving
those to x-ray spatial so in the geospatial world there’s fundamentally two different types of
data sets that we deal with um they their names collide with other domains but within gis we have vector data and
raster data the vector data is representing discrete phenomena things
like county boundaries points lines and polygons are like the types that we think about on the vector side that’s
our adobe illustrator for designers then on the um regularly gridded data set
side that represent continuous phenomena things in the geo we think about elevation rainfall
um any of like soil types those are many times better represented as as
grids and that’s the raster side of gis so what xray spatial is trying to do is
um really connect two different communities within data science which is one the
array computing community which is familiar with numpy and then the gis community
who are familiar with geospatial tools and sometimes needs spellings that are
more geared towards gis professionals so as an example um a gis professional
thinks of slope and um other you know say electrical engineers think of np.gradient
those are very similar functions but have it have different names and if we want i want to help the gis community
use these amazing tools from the more general python data science community
can you explain to us a bit about the history of the name and the logo for the project
yeah so the um the name is really an ode to the x-ray library um so xray is the
fundamental data structure that we use for the inputs and outputs from functions
um the logo uh sarah miller who is a great designer local designer here in uh
south austin um designed the logo for us um and that you know was a was a
a nice thing to have to add you know more structure to the project and be able to um you know have a sweet slack
icon and stuff like that so there’s a little history there um and shout out to sarah and sarah miller designs um who
does great work and um but yeah so the name is um based on the x-ray library and then also
spatial is a little on the nose in terms of uh what it does because that’s you know the main focus but um
naming can be difficult uh but right you know x-ray spatial is is uh hopefully
tells the story of what it’s meant to do in the quickly yeah i think uh i admire the name and i
also sympathize with the struggle of naming a package
that’s also like draws people in you know maybe that’s clever it’s challenging for
sure um can you tell us if there’s any alternative projects out there um that
might fill a similar void and how you differ from them yeah so there’s some great projects and
a lot of these projects are ones that we actively use all the time um so one of them is um
rasterio so rasterio is i believe was created by sean gillies and
it wraps um gdal to be able to provide raster
support for the python ecosystem it’s great and gdal is great and gdal
has has 20 plus years of development behind it and is a is an
awesome library um rasterio ties into x-ray also and that’s
a nice um a nice hook raster stats which is also from that same rash stereo
ecosystem and perry geo on github provides some nice zonal statistics
tools x-ray spatial also includes zonal statistics and the idea behind zonal statistics right is that we’re taking
one grid and we’re summarizing another by it so we may have unique values that
represent zones we could imagine polygons on one grid and we want to get basic summary stats
min max standard deviation variance for uh another uh grid raster stats is a
library on github that will also do that um using gdal and does it really well
so that’s some overlap with our library now the main difference is that is our interaction with the c level
so gdal and rasterio raster stats um they interface with c by wrapping up
c and c plus plus into c extensions x-ray spatial interfaces with c by using
the number compiler the difference there means that we can extend x array spatial within python
very easily so you can access all the core algorithms that are running that
are actually in say rash stereo or raster stats implemented at the c plus plus level inside of gdal you can access
that within python and then you can extend that and that’s the the main one of the main
technical value propositions of x-ray spatial is its interaction with the c level so instead of uh everything is written
you can extend that um you know without having to change languages
yeah that’s so cool and i feel like you already answered my next question but i just want to remind the audience
uh that you can ask questions in the chat so we’ll do our best to try and answer them at
the end we’ll have like a question uh session at the end but so what
technology is x-ray spatial built on i i feel like you’re already mentioning some of that stuff but is there anything else
you want to talk about definitely so um x-ray spatial
falls inside of the pi data stack in general so the core dependencies of
x-ray spatial include x-ray but also data shader for some of the rasterization
number dash pandas numpy and also spatial pandas which is a library that
came out of data shader also for vector geometry um headed up by john meese
that is another um library on on on the x-ray spatial readme we have a a graph that shows the
dependency libraries and also optional dependencies so just a shout out to a couple optional dependencies definitely
rest stereo so if if you do want to use gdal there’s integration there especially for
um input output say reading geotiffs and then also things like uh hollow
views which allows you to easily compose beautiful dashboards and visualizations in python
so we’re trying to play well with uh the whole data science um ecosystem within
python including jupiter notebooks and um so that that a little a little note on that
i mean yeah um i think you might be on mute
oops sorry yep i was on mute so when i want to type i just don’t want you to hear my mechanical keyboard
you know um yeah so thank you um so you mentioned a little bit before when you were um
moving functions from um data shader into x or a spatial that you
said now you’re the like a person who’s maintaining x-ray spatial who else
maintains the project and uh is there anybody else yes so we currently have about 10
contributors um eight actual listed contributors and uh we’re
definitely have been moving outside of the of the make path circle
contributors specifically for the focal statistics module um we’ve had some really nice contributions there
um in running kind of kernels over rasters and doing things like hot spot analysis where you can pull out
statistically significant areas of a of a grid and that has come from uh from outside
of the make path group and that’s what we want to see make path is sponsoring this project but we don’t want to see all the
contributors be from make path we want to have this be a well-supported project and potentially also move this into an
organ a proper organization as opposed to being inside of the make path company
github organization so there’s some things that i think we could work on there in terms of making sure that there’s more community involvement
and some of the efforts around also participating so yesterday over the past couple days we’ve been celebrating
international gis day and with in austin uh there’s a conference there and we’re trying to get
the word out about uh two gis professionals about x-ray spatial and
adding x-ray spatial to your existing toolkit so that um you know you can deal with
any situation that comes up um we’re really trying to frame this as um
as a really nice complement to existing gis tools so if you’re someone that uses
arcgis or someone that uses qgis you should look at xray spatial as a supplemental raster library that you
could use to solve a problem where you might not be able to hold all of the data on one machine and you need to
scale out to a cluster that could be a good use case for using x-ray spatial with das and being able to solve a problem that
you couldn’t do on a single workstation yeah so you mentioned gis professionals
but that’s like a big group so what communities are your users and potential contributors from
well they’re obvious there’s a big focus on python and so that’s one of the major first dividers in in the side is that we’re
we’re focused on the python community and making sure that the experience for um people wanting to invest in that
whether they’re beginners or their experts that they have good support uh in terms of modern package managing um
all of the the tools that we need to support contributions that come in um so there’s there’s that side and
it’s a real there’s a major focus on people that would classify themselves as
gis analysts these folks a lot of times are coming from
a deep domain expertise but may not have the software engineering background
to appreciate things like um you know c versus fortran
row major column major concerns um if we could take
tools like cuda and empower gis analysts with very
specific domain expertise say in conservation um to use gpu hardware to solve their problems
then that would be a big win for us and so that that is a little a little bit inside of where my head is at
and trying to support domain experts that don’t want to learn
um threads blocks and grids in cuda yeah and just to kind of like build a
little bit off of that i think it’s challenging like when you start kind of building in this like domain specificity
into your into your libraries that then it automatically makes it sort of exclusionary to other domains and so
like trying to make your library both accessible to very specific domains but broad enough that like other people can
come in is very very challenging um and i i don’t have a good solution
for this but i can imagine this is a problem you’re thinking about a lot yeah and and also not always just
thinking by analogy to some other tool where you’re just always thinking okay well this tool had that tool so we need
to have that tool yeah yeah um but the um but focusing a lot on the
scaling issue of helping people to scale this type of analysis to um
even just multi-core being able to uh provide the what the desk bag provides but just with a
spelling that’s um familiar to people you know within gis um is one of those those stories we’re
trying to support yeah yeah cool very cool so can you tell us a little bit also if there are any
diversity and inclusion efforts that x-ray spatial is working on or implementing or interested in doing
yeah um so i would say kind of like the first thing there is that it is open source
right so that’s just at the beginning of being able to have a a code base that anyone on github can come to and read
and contribute to starts that process of um building a inclusive and diverse community
i think that we we all want to maximize the value
um an open source project to the world and to and that means that
if there’s people that are coming to the table and aren’t able to contribute for
some immutable characteristic about who they are or what they do or where they’re from then that hurts
that value proposition of delivering uh world-class tools that are open and free
um i think that the that dealing with issues where people are either are
excluded or there’s um community problems within x-ray spatial starts with a code of conduct
and having a reference point that you can look at to make an issue not subjective
so one of our the goals there is to make if there is a problem it’s not a subjective thing it’s
something that we can point to a document that’s not one person but that
is an agreement between all of the participants and say something that happened in the project
was wrong and it was wrong because it violated a code of conduct that we all agreed to so that’s one of the at least
defensive tactics that that we’re using which is very common among projects
to be able to have some of that foundation god forbid there is a problem where
someone is excluded for um you know some ridiculous reason and
that’s so that’s a little speaking to that but the um that’s what we want i mean we want
people that we’re addressing very difficult problems there’s a lot of different different
cases and it requires a very diverse group of minds to solve those cases
so hopefully we can do more are there any suggestions that you’ve seen in other projects that when you
think about x-ray spatially say you know what we had another guest or another um look at another project as being
particularly exempt exemplary in the area of inclusion um i don’t have like a specific project
i something i’ve been thinking about is like trying to create some sort of onboarding or mentorship program for
like new contributors so like you know if you see somebody is new trying to come up with some sort of like
you know way to make sure that they uh like are immediately included by the
community and like have a pathway to like become a part of the community if
that makes sense um yeah so that’s something i’ve been thinking about and i know it involves
like a lot of ask from devs and you know on my project we i don’t have a lot of developer time so of
course there’s tension there um but it’s something i’ve been pushing for um yeah so melissa do you have any
thoughts that tony is mentioning pedro in the comments and kendra was part of uh open
source directions a few episodes ago and they they definitely do have like a whole focus on documentation and on
like bringing people in through information but also through that contact as mentoring and onboarding and
having people who are open to having that discussion and and um
just making people feel comfortable contributing at whatever level they are like
meeting people where they are if they want to contribute and and bring them in the project so
i think those kinds of active um um initiatives like
code of conduct is like a static thing like it’s in your community and it makes
it better for existing but you don’t have to necessarily do something it’s just there
but having those active uh initiatives is also extremely helpful uh especially
if you feel like you’re your project is missing people is missing some group or
which is what happens unfortunately many many projects yeah that’s true thank you so much for
mentioning kendra tony and melissa like highlighting some of their work i really think they deserve like humongous props
they had so many sort of they had so many different initiatives you know it’s not just one thing it was
just like from very small to very big it was really impressive some of their work
it’s i think it’s really interesting also like in the in this new era of remote work where we have a new focus on
remote work how that new culture influences inclusion and diversity in open source and so
like a couple of things jump out as i’ve been thinking about this one is that it’s nice that everyone’s remote and we
can be on an equal playing field in terms of seeing commits and pr’s come in um but also sometimes the remote nature
of not seeing people face to face means that people behave in ways that um aren’t as nice
uh because they don’t really feel the impact of the statements that they make and so those are just some of the things
questions in my head that i’ve been playing around with in the in the new remote era yeah i think also something i want to
mention is that uh i think you know because we’re all remote and kind of isolated right i think that means
inclusion is extra important because if somebody’s already feeling sort of you
know on the outside and then they get some really rude comment or uh you know
degrading comment and a pr it it hurts more right or it hits harder
and so like being a truly inclusive community i think is extra impactful in
a time like this and also like there’s a tendency for
people like it depends your life situation might influence how you see things so
for example for me i know that uh being remote is extremely hard and much harder
than it was before because i have a kid and he doesn’t have school anymore so i have no free time i have zero free
time which which if i didn’t contribute to an open source project in my day job i would not
contribute at all so we are missing a lot of people who are having like family obligations have to take care of loved
ones of children of elderly so it is hard um on on many people so yeah
this is just something uh just a little touch of
reality yeah i think it’s an important point melissa i mean the fact that there are
people who are not being included right now based on the um
the work situations that are existing is really a problem right um
so yeah so anyway i think we can move to the project demo
i don’t know this is a great canvas though um so yeah i think we can go to the
project demo great some of the cool features of go ahead and share my screen here yeah so while you’re getting set up
uh we would just like to take this opportunity to thank our sponsor kwonside for sponsoring this episode of
open source directions one side creating value from data so whenever you’re ready brandon
great so um are you guys able to see my screen now you should be seeing the x-ray spatial github yep
um so so first we’re looking at the um at the github page for x-ray spatial um
we can see some of the uh the installation let me i’m gonna make this just a little bigger so folks can see
and some of the descriptions here we have a list of the supported spatial functions and their supported inputs
so when we look at x-ray spatial we’re doing raster processing on basically on
x-rays which wrap numpy arrays but they can also wrap dash arrays and
coupli arrays and then we also want to support a dash gpu
input and so this first lists the spatial uh functions that we support things like
classification surface analysis with uh curvature and slope
multi-spectral analysis with things like ndvi and a lot of the different flavors of ndvi
you could use these tools on top of landsat data um and depending on which flavor of
landsat you’re dealing with you may use different bands the bands might be a little different but you can
still use these tools on them so ndvi we have a bunch of proximity tools
so these are about calculating distance or associating features
with uh with their closest neighbors um direction would allow you to do
things like say what is the direction to the nearest location so i want to know in 360 degrees
um so let’s take a look at a couple of these tools in action to do that i’m going to come up to the
to the examples area and go over to the user guide
i’m just going to click on this first one which is the surface tools
it’s a little description of what we’re what we’re trying to accomplish with these surface tools things like hill
shade slope curvature and aspect aspect for folks that may not know is the um is the direction that the slope
is facing well curvature is the second derivative of the of the elevation
and slope is the first now you’re bridging the gap between uh non-gis people and gis people because
there’s no way i would have known either of those things exactly exactly and so the um and you can imagine view shed in
is basically like one step raycasting where all you’re
you’re we’re connecting points that are in line of sight of each other but not continuing the the cast after one hit
um so when you’re using these tools you need a grid to work with and so we
provide a uh just some procedurally generated terrain to play with
and this is um just some purlin noise that we’re playing with and we’re using data shader here to
apply a color ramp the elevation color ramp with a linear stretch we can start to see how we can use
data shader with x-array spatial to interactively do analysis and see the
output of that analysis as we go so we’re taking this same
elevation data set and now we’re illuminating it using a light source at a specific altitude
and we can composite these two images together using data shader using data shaders stack so here i’m
taking a an illuminated terrain and stacking it with a color mapped
elevation terrain slope where say we have the the use case
where we’re worried about avalanche and we’ve defined a threshold for avalanche based on slope
we could take the slope of the terrain use numpy to threshold the slope
and identify those areas in red that we believe might have
additional avalanche risk curvature
you know you can think about you know second derivative acceleration and some some other use
cases i’m not going to go too deep into curvature but know that this is here
and aspect another um another surface tool on on view shed let’s look at view
shedding view shed we’re trying to we take an observer location and we want to
highlight all the pixels in the numpy array that have direct line of sight to this observer
and so we’ve put a kind of a normally distributed little chunk of data in the center
and we run view shed and we can see all the pixels highlighted in
red that have direct line of sight to our observer and this works you can imagine this being used with like cell phone towers
where you’re trying to look at the service area for for a cell phone coverage so you can use
multiple observers here and specify exactly what the altitude of those observers are and what the altitude
above what you’re trying to line up with is so if you’re trying to make sure that um
you can see 10 meters above a target those are some of the parameters that you can specify within
uh within hillshade i have a question about this image actually so you have the um like there’s like a
dark gray behind this conical feature um and then there’s white and i’m assuming white means that you don’t have
a direct line of sight so what is the dark gray just like extra bad line of sight because you have something in
front of you or um it’s actually just illuminated with hill shade to give you the impression that there’s height here
okay so you can kind of imagine that this this observer and it might be a little hard to see it’s orange right
here is uh is not able to see anything behind this little little uh
mountain yeah okay i see so you have like both the line of sight data and this hill shade uh like overlay um so
you can still see the features in addition to the data of the line of sight
yeah and the output grid is actually not only the line of sight but the um
angle to that spot okay so we see we see that the that the
actual um red here is lighter as opposed to darker here as we get to
the top it might be kind of hard to see but um so that output grid includes the direction that the person would look to
uh to hit that point oh interesting okay cool um and that’s helpful for thresholding
right because you want you want to maybe filter by that afterwards um so this is just an example of kind of
on this terrain if we were an observer on this slope what could we potentially see here
and we have some references and we you know we didn’t uh really invent any any of these algorithms we’re trying to take
these items and make them fast using number
very cool um i have another like minor question about this with your grid data
um does the data always have to be a regular grid or do you also support non-regular
grids um right now the um it’s regular grids right
and we need to have a good abstraction for how we deal with you know polar coordinates and we don’t really
we don’t dive in too much into um like the curvilinear
rasters but the rasterization process of getting of
getting points lines polygons try meshes quad meshes and other rasters into a
shape that’s um usable for x-ray spatial can be done using data shader
okay so so data shader can handle for instance if we think of the sentinel 5 european
um satellite data that comes in curvilinear coordinate system
um and that it you can aggregate to a data shader canvas
within data shader and now you’re you’re
you’re dealing with a um something that can be aligned with a
rectilinear uh grid okay this is extremely cool i’m just
like having so many great questions coming through so ah yeah i love this demo it’s great
anything else you want to show brendan um sure just one other uh
is that zonal statistics tends to be one of the very high value tools within the
gis ecosystem this is where we’re uh and let’s just look at a quick example of that and how
you would do that in x-ray spatial which is a high-value tool we have our same terrain
and this time we want to imagine that we’re hiking across the terrain over um
five or six different days and we want to look at the summary stats of the terrain for each one of our day segments
we can use zonal stats to do that so here i’ve rasterized um my
trail segments using data shaders line feature or line function on top of a canvas
and then i’m going to use x-ray spatial zonal stats to then um use
the the trajectory data to calculate the summary stats of the underlying
elevation data so here we have one record per day of our hike
and we have the mean elevation maximum and all the you know the the summary stats that we want for that
part of the hike the x-ray spatial also allows you to define
any custom zonal stats functions that you want so as you want to extend this for your own purposes
we will call this function and with the input zone
rasters that you need to or numpy arrays that you need to do your calculation so you can do custom zonal stats for each
zone
it’s really cool and and i bet that has applications well outside uh gis
community like i can see using that for planning a hike or for you know like i
don’t know planning like a running um training
plan you know like yeah i mean you can see that being used by a lot of people and you can see like
it’s it’s a nice api and i just think it’s really cool oh thank you yeah it’s very cool i would
plan my run with minimal elevation change [Laughter]
uh being a midwesterner so let’s not be on the mountains that’d be
great you know or just stay at the same elevation the whole time [Laughter]
um i could certainly go on about zonal stats for for uh for a whole other podcast probably but so if anyone has
questions about this please feel free to reach out to me and i can i can help answer any specific questions on zonal
stats that looks really really nice and i
really like the examples and i just want to know if you have like you can definitely if you have any other
stuff to show uh you can definitely go on but i just want to know like do you have an idea of the road map uh the liza
had for x-ray spatial what are your plans for the future where is the project going and where are the
priorities of the community right now yeah so um right now
we have a couple of different kind of road map features one is better documentation
building making sure that the documentation is robust and that we fortify the library for all the tools that we
already have as opposed to adding a lot of new tools that’s one of
the themes in the short term roadmap we want to build out a better x-ray
spatial documentation site we have one now but it’s a it’s an old build of the sphinx and we we need to
update that so documentation is very important here and then our main story now is going to
be scaling the tools that we have so i’m going to zoom out a little bit just so you can see this full table but we’re
filling out this table so for instance the aspect tool of what direction a
slope is i want to make sure that so that works with uh coupe gpu arrays right now but
we want to make sure it works on a gpu cluster and that it works across just a standard
cluster and if we look at um if you if you don’t mind i could actually show some code real quick of
what that looks like yeah and how easy easy that could be for someone to help contribute
yeah here here’s the slope function and if we go to the bottom of the slope function we’re going to see
the public slope method here so
all of the functions take in x array data arrays and you can assign the output a name
that’s part of x-ray but what we do in each one of these functions is that we just have to try to handle
these four cases so if the x-ray object is base is backed by a numpy array we
run a numpy case with coupei a with a coupe backed um dasquaree or a
dasqueray with a numpy backed desk array and so as we go through this this is a
slope is a good example of how if you wanted to contribute one of these functions for an another
tool which lacks it that you could look at slope as an analogy and copy some of this code
so most of the code for instance if we look at what it’s like to run slope across
a das coupey um array we can just look at this
function here and maybe take that and and use it for um the the hill shade which doesn’t
currently support hill shading across a gpu cluster so this is this is going deep into the weeds here but this is
what x-ray spatial looks like on the inside is that we have um
just a bunch of of of number jitted functions
that apply to either the gpu so this is the gpu of slope or the
the cpu version which is a a number function for running horn slope
on a cpu so quick you know
into the into the weeds here um with with x-ray spatial but slope in particular is a good module that if you
wanted to make a contribution you could use slope as an analogy um to extend things
yeah i think this is great i mean knowing knowing that you need this right and if somebody wants to contribute that they
know a good robust module that exists to model stuff off of i think is really helpful right and that’s like a nice
pathway for a new a new contributor for your project you know and you mentioned you wanted people
outside of make math so it seems like this is a nice option you know um is there any
um related to this a little bit i’m going to ask you a little bit about contributors now um do you
um you mentioned kind of doing community work outside of your um
you know to the gis community um have you been successful
in getting gis contributors to the project um and how have you been doing this sort of outreach for your project
so we really i think a lot of the the we have been successful in getting um
and getting some good contributors um from outside of make path um not too many i mean the
project is is fairly small right now but the the majority of the contributors are
outside of make path um and uh now this is a very small sample
because we only have eight official contributors but the um uh
so making sure that on the outreach that we’re telling letting people know the
the problems that x-ray spatial solves in those domains um so i’m really thinking here there’s
people that don’t classify themselves necessarily as gis professionals but in fact do a lot of spatial analysis
especially in academia we’d be very interested in um getting
more contributions from um just different domains in the in
academia that use things like our hot spot tool to
to look at to look at statistically significant hot spots on an area the
the atx gis day that we were participating in in letting people know that this exists
most people actually just don’t don’t know that this exists at all right um and getting that word out is really
helpful and so you know thank you guys so much for for having me on here on this because this is really helpful for
x-ray spatial to to get more contributors yeah are there any um like additional features that are in the gis community
that you haven’t built out yet like you mentioned that this is a library of you know sort of gis functions or like
functions you know spatial functions right so like what uh you know additional features are you kind of
looking for and also like just related to this just to complement the question uh are there
like uh i know that for example in academia it’s very common that you use commercial project products or maybe an
industry you have like a different closed source products that do the same thing do you have like a a plan for
bringing those people in or you know like doing outreach and telling everybody that there’s a free open
source version of the same thing they’re using yeah that’s a great great question and one
something that i’m trying to um frame x-ray spatial with is that it’s
even if you’re a dedicated user of a proprietary product there’s a role for open source and python and
x-ray spatial within your toolkit and um trying to break down a little bit of
the us versus them open proprietary dichotomy and seeing how
simple functions that can be added to um to link those tools together to ultimately have get analysts to have a
better overall set of tools so one of the things that we we do here is that the most popular gis proprietary
software package is esri’s arcgis desktop um we arcgis has python integrated into it
and they have some really nice functions for instance raster to numpy
so within the arcgis community they have access to be able to convert their proprietary
data formats into numpy arrays this is on very short
term roadmap we are adding and utility functions that if you have the arc pi library
installed on your system then we can read arcgis proprietary
rasters just using this raster to numpy and numpy to raster
that is is very low-hanging fruit for x-ray spatial and adds a ton of value
for anyone that’s in that ecosystem and and needs these tools to integrate yeah i think the um interface between
proprietary tools and open source is very interesting right like i think there are a lot of specific domains
where you’re required to use uh you know like to do any work in it you
have some required requirement of using proprietary um software and so you know trying to
complement that with open source so that you can still do your analysis publicly um even if the way that the data is
generated this may be proprietary still has a lot of benefit right um in
my case you know nuclear engineering we we don’t necessarily have um like uh proprietary in terms of like
um business proprietary but we have a lot of export control tools so like there’s nothing i can do about that
right like i can’t not use export controlled stuff in in some cases
so i have to do that so doing my analysis publicly instead is like a way to kind of be
open source and live through an open source ethos so it’s nice that you’re building that for the gis community
yeah as long as this is so easy it’s so easy to integrate this
that it doesn’t really at least for me have to become like a philosophical issue of integrating with a property
piece it’s just easy low-hanging fruit that we can do to um to empower those people that are
already using arcgis and and want to use it
well thank you so much brandon i don’t know if you have a final uh notes
things you want to say before we move to the questions because we do have a couple questions to answer yeah i mean just um uh just so happy to
be here and um thank you guys so much with open teams and all the all the questions coming in
yeah so we’ll go to user questions um we have a couple ones so first as you are
building new communities around an open source project what are some other open source projects that provide provide you
with good models examples um to understand how to grow and support your emerging communities
um i learned a lot from some of the communities within python desk in particular desk
and bokeh both of these projects helped me learn
what it is to contribute to open source and how to participate in an open source project
uh and i would really like to encourage anyone that’s interested in contributing to x-ray spatial that there are
needed contributions on every skill level so if you’re new to programming
altogether and are interested in the domain that x-ray spatial is addressing then there’s a role for you in the
project in terms of helping out to define features to test features to help
with documentation and other things if you don’t want to directly contribute to say gpu code
so there’s just letting people know that
there’s need at every level everyone can be a valued contributor to x-ray spatial so please you know come by and uh and
check it out melissa why don’t you ask the next one and i’ll ask uh my last name yeah
so just a second one is does make paths have any specific ways that they support a remote working culture
um definitely so make path is fully remote um we started fully remote from
the beginning and in doing that there’s some unique challenges um most of it’s actually very smooth uh
but we do things making sure that we prioritize um using video making sure that we’re
having a lot of conversations and prioritize communication over um
you know extensive you know documentation on certain things so the there’s unique challenges to
remote work but i think it’s also making sure that people stop at some point um and
everyone working remotely i’m i am uh fall into this also where sometimes there aren’t clear stopping points for
folks um where you know there isn’t a nice kind of work-life balance for people that are um
always in their office because their office is their home so we like to also make sure that people are are taking
care of themselves in terms of uh not working too much but um
i really like the freedom that remote work gives you and i’m not a big fan of a commute so i really like that
yeah okay so my last my qu the last question is a question for me and it’s related to um when you were talking a
little bit about how x-ray spatial came into existence um because you had all these very specific um
uh spatial uh functions in data shader and so i’m curious so i think a lot of
pro projects struggle with modularity and not becoming too bulky as a project um
and so i’m curious like how did you plan this process uh where you spun out some of those functions and deprecated them
from data shader and created this new library and was there growing pains with that and can you talk a little bit about
that because i think that’s very interesting oh yeah um so there there were some
growing pains there was certainly um uh just being able
to deprecate that code from data shader and making sure that those users were
supported and anyone that that was using those functions including downstream libraries from data shader
holoviews is one that does wrap data shader for for certain things and there was some risk there of breaking other
people’s apis uh i actually was trying originally with
x-ray spatial to implement polygon rasterization and then um
it just kind of spun out and i think that saying being able to figure out what to say no to
in terms of features is important so we’re only focusing on
um number eyes raster functions if it doesn’t involve um if it involves
for instance geopandas and manipulation of vectors not going here that’s another library
and that’s having a clear mission statement of what you’re trying to to do is
important so that you can evaluate new features against that mission statement and decide whether they’re they’re valid or not
um answer your question midikin you’re going to come on that i know i thought
that was great i mean i think this is like you know a great sort of conversation that can you know continue
in other uh places outside of this uh webinar but i thought it was very interesting and uh
thank you because i think it you know it it’s a challenging problem i think when you start doing these sort of um
uh when you start spinning out functions from one library into another um you know it’s a very interesting uh
story so thank you for sharing that oh thank you guys yeah and i definitely agree i think like
having a road map seems like such a healthy thing for any project to have and and
it is hard when you see projects that don’t know exactly where they’re going and that that’s what happens is that you
end up absorbing too much or not being clear on what you want to do and then you end up doing a lot of things that
are not your main priority but just because you can’t say no so yeah i
absolutely agree with that and i think that’s a very healthy way to organize your project
um okay so uh now it’s time for our world famous rent and rave section
because we’re nearing the end of the webinar so each of us is going to get 15 seconds to
rent or rave about whatever topic we want so brendan do you want to start
um i guess uh i would like to rave about number i’ve been learning
um about cuda programming i’m still a novice but i’m getting up to speed and i’m absolutely blown away by uh what the
power of number and how easy it is to write what is very complex code on the on the c side so i
would like to rave about number and how impressed i am by number
okay well i’m in a rant um you’re so positive so i have to compliment you with something frustrating which is i am
really annoyed that christmas music starts so early like why i know we started this webinar talking about mariah carey but she’s like an angel so
it doesn’t count but why does it start like this year it started right after halloween i don’t need two months of
christmas music i’m not that person stop inundating me with christmas music stores and other places
annoying i don’t know about you but i’m gonna kind of do the opposite because i’m gonna rave
about christmas we’re almost there because i think i i have like an idea i
i have a theory actually that i think people just want to rush the end of the year and see 2020 done
so we can’t wait for christmas and for new year’s and to forget this whole year that has been like
so crazy so that’s my theory that’s why it’s starting so early this year because you can’t spend being in 2020 anymore
i mean i think we could have a much longer soapbox rant about how much 2020
sucks yes yes and um i um i was able to humanely catch a
chipmunk uh this summer out of my sister’s house and i brought it down to release it into the woods and i played
it the the chipmunk christmas song the entire way see if i could get
you know a single singing chipmunk that was my one opportunity didn’t want to sing but you know that was that was a
very crit that was my first christmas song of july i don’t know why you did not introduce
yourself with this amazing that you can mainly got a step back i love chipmunks oh my god i have some like woodchucks
that are living in my front yard and i record them they like chocolately run if i could have chipmunks and woodchucks i
would be like a woodland paradise you know oh so cute
well i think that’s an awesome way to end our conversation and
this is all the time we have for today so thanks everybody for watching thanks brandon from being here we loved it i
think it was a great uh way to showcase x-ray spatial uh thanks everybody for listening you can find us on twitter at
open teams inc and at ai brandon where can people find you an
x-ray spacial yeah so you can find x-ray spatial at um github dot com forward slash make path
forward x-ray spatial and also check out the make path dot com forward slash blog
so you can see some of the use cases that we’ve uh we’ve shown on on using x-ray spatial
great so if you liked what you saw today please go to our youtube channel and like and subscribe to see more of this
content and i’m not going to announce the next webinar because we’re going on a holiday break so this is the last
episode of 2020 and we thank you all so much for being here uh thanks again for
the folks in the chat also and that’s it that’s what we have for today
thank you thank you
