About
Jupyter Book lets you build an online book using a collection of Jupyter Notebooks and Markdown files. Its output is similar to the excellent Bookdown tool, and adds extra functionality for people running a Jupyter stack.
Transcript
[Music] hello the internet welcome to open source directions hosted by open teams
the webinar that brings you all the news about the future of your favorite open-source projects my name is Henry
battery and I’m the growth marketer at open teams and I’ll be your host open source directions co-hosting with me
today eight years hi I’m MIDI in and I’m excited to call this episode of open source directions
I’m a postdoc at the University of Illinois at urbana-champaign I work on a project called YT and I’m based in
Urbana Illinois and I’m Chris I work at University of California at Berkeley and
I primarily focus on the Jupiter project and a related project called binder thank you very much so before we go into
the meat of this episode we have our famous tweet of the week section where each of our panelists presents a tweet
that they’ve been enjoying recently so Chris you’re up first yeah so the tweet that I found probably
unsurprising is related to the recent kind of global concerns around that corona virus and I noticed that a fellow
Syfy PI data contributor guy ala bucco and several of his collaborators as well
recently shared a link to a little web app that they built using I think plotly and scikit-learn and a couple of
different pieces of the Syfy ecosystem the visualize is the latest data about
what’s going on in the world with the the kovat outbreak and you can include some projections and some modeling
efforts and things like that so that I was kind of a cool way of seeing the Syfy community coming together and giving their treatment to this awesome
and if you want to find this other link to the tweets will be in the chat and making your op Nick’s okay so mine is
related to animals because we all need a little bit of joy in our lives right now
so maybe some of you have seen the Shedd Aquarium has let their penguins walk
around because the aquarium is closed so they’re letting them like walk around in the general area around the aquarium and
so this particular tweet is a video of a bunch of penguins hopping down stairs like they’re going from one floor to
another and they don’t walk downstairs there’s like this very adorable plopping noise as they come down so please enjoy
awesome so yeah my tweet of the week which will be posted in the chat so essentially open-source ventilator
Ireland there’s this movement towards creating a ventilator design and then
trying to figure out how we can produce that at a very very low cost given the corona virus and the pandemic that’s
actually happening at the moment I so I thought this much weight of the week because I I love the idea that everyone comes together and the open source
really the open source ethos is probably one way that I think we can all collectively come together remotely and
work towards really solving this problem and we all know I just watched a video that in New York they’re calling for
anyone who’s basically got a ventilator but donated and helped people that because it’s going to be it’s in high
demand and it’s very little supply considering everything that’s happening so so much share that all right well now
that we’re done with the tweet of the week let’s dive a little bit deeper in view for the book so derivative work is
an open source project that lets you build an online book using the collection of Jupiter notebooks and
markdown files it’s a simple good book down tool and it adds an extra
functionality for people running at Jupiter stack so you can find jupiter book on github at github comm slash
jupiter slash jupiter – book it has around 669 stars on github and about
1932 downloads a month on pipe I so so Chris can you tell us now that we have
this nice introduction to Jupiter book what was this president why was this project started and what need does it
feel yeah sure so originally Jupiter book was started at UC Berkeley and we
needed it in order to provide open notebook based on learning content for some of the larger classes that we have
at the University so in the last couple of years Berkeley has revamped a lot of its pedagogy around data science and
data analytics and particularly around introductory data sign and data analytics we wanted to use
largely an open source stack that is built on top of Jupiter infrastructure
and the kind of broader sigh pi PI data community and we had a lot of content that was written in notebooks but wanted
a way to present it to students in a way that felt a little bit more familiar a little bit more like a traditional book
that they could sort of quickly glanced through and browse so for this reason we created Jupiter book as a way to very
quickly convert a collection of our notebooks into static based HTML site that we could just send a link to
students and then they could quickly look through the content and then you know grab a notebook if they want it to
really dive deeper into what was inside yes so it sounds like you’ve sort of built it and Venus bye-bye other
technologies which was very prevalent in his space so could you explain a little bit of the history of the name and logo
of the project yeah so I mean I think that essentially we were trying to think
of what would be a good representation of a collection of Jupiter notebooks
presented in a more traditional form and so it seemed like a book was a good way
of thinking about that so if you think about each notebook as an individual page then the collection of notebooks is
a collection of pages and it’s perfect
to make puns about so yeah very funny
yeah so a lot of really great projects out there that aim to help you kind of
stitch together collections of content files and then show them as like a holistic unit probably the most
well-known one is a product that comes from the our studio community called book down and this is a really excellent
tool in the our community in many ways it’s kind of in my opinion one of the gold standards of creating like
computational books and it definitely inspired a lot of the creation design ideas around Jupiter book which tries to
serve a similar kind of role before the Jupiter community and the king you know building around the Jupiter stack
they’re also a couple of really interesting services that people run for hosting collections of notebooks probably the earliest one and the
Jupiter world is called and be fewer which is a service that the Jupiter community’s been running for
probably like six or seven years and it just likes you really quickly render a notebook it’s hosted on github and then
browse it as like a static HTML page and then or a recent one that I just recently learned about came out of the
FAFSA a I project and Jeremy Howard they really they recently launched a really
interesting project I think it’s called fast pages that uses github pages and
you have actions to quickly allow you to generate a static blog using Jupiter
notebooks on github and I’m looking forward to checking that one out more – okay that’s awesome yeah thank you for
sharing that and one thing that I’ve noticed last week with Pi janitor they have were inspired by the our community
in some of your day field and it seems like you are also inspired by something the our community appeal to so it’s
interesting to see that pattern but on the topic of that so what technology is
it built on your project right so there are two different answers to this
question and I can get into more details about the details there later on but the
the short version is that the current version of Jupiter book is largely a combination of two things one of them is
a Python API and a little lightweight Python library that uses a project
called env convert which can convert Jupiter notebooks into a collection of other kinds of output formats in this
case HTML and a project called Jekyll which is a ruby based static website
generator but allows you to stitch a collection of HTML pages into a static website and the current version of
Jupiter book basically just wraps those two pieces and then gives people a command line interface that they can use
to quickly generate books from their content however it’s I say it’s a
complicated answer because we recently received a grant to work on the Jupiter book project and a few related tools
from the salon foundation and we’re in the middle of rewriting most of the backend of how the Jupiter book project
uses the underlying infrastructure and in particular we’re gonna try to utilize Sphinx which is another Python project
and documentation and stack for Python and we think that that’s going to both allow
us to have more high quality content and include things like citations and equations and cross-references to tuck
it into later as a lot of standardized our toolkit along some of the other standard tools in the in the Python you
could system can you tell us who started
it you kind of hinted at this a little bit but I’m very curious yeah so I think that well it’s at the Henry’s points the
inspiration for a Jupiter book has come from lots of different kinds of sources so in a sense you know it was started by
a collection of open source projects most specifically at UC Berkeley there’s
always been kind of a Jupiter team at UC Berkeley that’s been part of the Jupiter kidney to be more broadly and we helped
the university develop tools deploy those tools maintain those tools for some of the pedagogical and research
news cases at the University and so we had initially developed Jupiter book to serve some of the larger data science
courses that we have that’s great to hear yeah so exporter minded listeners
to that right now we’re talking about Jupiter book and in the roadmap discussion we’re going to go more into
detail about where you can keep contribute and how you can contribute but if you are hearing things that are
interesting you and if you do have any questions then please post the questions in the right-hand side of the webinar
screen and we’ll get to your questions in the Q&A section at the end of the episode so please ask anything that
comes to mind anything is interested interesting you or anything related to Jupiter book just make sure you list
some questions so you talked about yes so that we’ve we know who started it that’s also that’s great to hear that
they came out of the university and and you’ve really been driving this could you tell us who maintains the project
yeah so I think that the most hands-on maintainer over the last say year and a
half or so which is about the amount of time that project has been around has been be as a part of the Jupiter team at
Berkeley we also got a lot of really good help early on from Sam Lau who was a student at UC Berkeley a year ago and
Elizabeth you pray who’s up in Montreal and it’s also contributed quite a bit it
has now like a broader kind of a relatively small but more diverse community of contributors that just help
out here and there and in the future as I kind of alluded to earlier we recently
got a grant then it’s going to partially join together a couple of different python-based book projects that
hopefully also grow the contributor base of which bitter book project and the process cool I’m so can you I’m curious
like you’ve talked a little bit about the origins and your book kind of starting in academia and some of the
collaborators you talked about also sound like their academics can you tell us what communities the users and the
contributors are from yeah so I think that originally it was you know people who are teaching classes and that’s
certainly still a really common user archetype that I see easy Jupiter book
but really I mean to provoke is a it’s a communication tool and it turns out that
there are lots of different use cases where people look wants to just share collections of notebooks in a way that
looks like an online kind of interactive book and so a lot of people that do you
know the run data science projects and want to just share the results with people and a little bit more of a familiar form people that want to bundle
notebooks as a part of like a scientific paper or research analysis have used
Jupiter book as well and know a few different publishing pipelines or looking into using it as a part of their
publishing tool chains so I think that that that’s becoming a more diverse group and just really quickly one other
thing I forgot to mention so on the Jupiter book repository on github github has these dependency chains that it
builds so it can tell you like what other projects depend on a given project and actually if you click on that you’ll
you’ll basically get a list of all of the different Jupiter books that people are hosting on github I think it’s
something like six or seven hundred different books and it’s a good way to just get some inspiration for what are the kinds of things the people are using
the tool that’s also yes sounds like you really do it out of community who’s really taking this and grabs it and
taking ownership and sounds like the contributor base to is a lot recently so sort of leads into my
next question has the project’s been participating in any diversity and inclusion efforts yeah so I think that
Jupiter book as a sub project within Jupiter more broadly has certainly been
making a lot of inroads into this Jupiter itself is like a really large open and attempts to be inclusive
community and it’s done a lot of different work to try and improve these these pieces of its community over the
past couple of years it’s always a work in progress and there’s definitely still a lot more that the Jupiter community
can and will be doing but we try to you know for example have all of our meetings open to the general public and
you know meeting minutes posted online so those can make the timezones they’re there a lot of the teams that I work
with we alternate our team the schedule of our meetings to be more
aysia friendly or America friendly or Europe friendly we also have a few kind
of specific efforts in diversity and inclusion so we recently applied for one of the czi essential open source science
grants the eos s grants which we luckily were awarded one that was with jupiter
hub and the binder projects which are kind of you know related to although not the exact same thing as jupiter book and
one of the main goals of that is to be able to pay for time for people to work on the project and we’re gonna
prioritize that time for people that probably wouldn’t be able to contribute otherwise so right now that’s gonna pay
for a woman in Romania in georgeanna to do some work on the littlest Jupiter hub
and the binder project as well that’s great to hear yeah it’s great to see that it seems like you’re really participating and um you know and you
have it having goals and at least a roadmap to be able to expand that and make the contributor base more go-to
first which is great to hear so now we’re going to shift into the project demo where we’ll get to see some of the
cool features of Jupiter book and how it works so while Chris is getting set up we would like to take this opportunity
to thank our sponsor icon site for sponsoring this episode of open source directions Quan site creating value from
data so you ready Chris take it away okay can you all see my screen yeah yeah okay I’m
gonna work from a couple of slides and then I’m also going to hopefully pop out and do a few demos and we’ll see how
that goes okay I don’t really need to do my own introductions anymore but really
quickly I want to point out some of the faces down at the bottom so in the bottom left is Elizabeth Duprey and sam
lau and as i mentioned both of them did a lot of work on the earlier iterations of jupiter book and on the current
version of jupiter book and on the bottom right that’s John Sturges ski and Greg pepper also PIAA nu and Australia
and a PI at Northern Arizona University in the States respectively and they are sort of key contributors on the next
version of Jupiter but I’ll talk about a little bit and so really quickly just to
motivate the reason that this existed in the first place was the general
inspiration for how Berkeley has been approaching its courses at the University which is that they want their data science education to be for
everybody at the University and by that I mean students who aren’t going to be you know computer science majors and
sophistic majors but also for students who are going to be you know philosophy majors and English literature majors and
Poli Sci majors obviously they don’t need all of them to take like a full four years worth of statistically
rigorous courses but they want all of them to be familiar with the concepts of data analytics and statistical thinking
and whatnot the challenge is that at Berkeley you know it’s a large public
university and if you say oh we’re gonna offer like a data science course for literally anyone who wants to come without any background and you know
coding or or statistics you get classrooms like this so this is the largest well it’s not even really a
classroom this is the Performing Arts Hall at UC Berkeley and it’s filled to capacity with students for their
introductory gain an eighth course which is a course that utilizes the Jupiter
stack very very heavily and so a lot of what we have to do is try and build technology that can facilitate the
learning for this very very large and very very diverse and heterogeneous student population some of whom are
familiar with coding and Python and whatnot but many of whom are not and so I’m really the goal of the course
is to teach concepts and give them some experience with interactive computing
using an entirely open stack and also to partner with other open projects and
open communities in doing so but what we didn’t have the ability to do was to
display a kind of notebook contents and a more familiar or accessible form particularly for the students that
weren’t already comfortable with like spinning up their own Jupiter notebook server and whatnot and the other thing
that we wanted to do is try to strike a balance between like full interactivity
with a live Python kernel that you would get and like a jupiter notebook or jupiter lab versus like a completely
static website that you would get if you were just you know rendering some some HTML and like a traditional blogging
platform and so really jupiter book is trying to strike that balance between the two of them and give the author the
ability to choose how much interactivity they want to bring into the reading experience okay so that brings me to
jupiter book and just to repeat the tagline for the tenth time the goal of
jupiter book which will allow you to online book with jupiter books and with markdown that down there that link
that’ll send you to the current version of jupiter book it looks like this imma
be the easiest way to describe jupiter book is just to show you what builds a Jupiter book looks like and this is
actually a page I can probably just go to on my own so if you go to inferential thinking com you’ll actually see the
generated book for that dating aid class so all the material is hosted online it just exists as a collection of Jupiter
notebooks on github and really off the bat the first thing that you’ll see is that it doesn’t really look that
different from what a lot of other websites do you know it’s got like a left table of contents it’s got a right within page table of contents but if you
look at the contents of each page itself you can see that there was a notebook under the generated all of this stuff so
this page itself is actually in a Jupiter notebook on github and it was rendered into the static HTML form with
all of the outputs populated and whatnot using Jupiter book and so there are some things in here
that you can start to play her out with and give the authors a little bit more
control over like what kind of interactive interactivity they want to insert I don’t know that the inferential
thinking textbook makes use of this but I’ll show you an example of this later on but for example if you want to
include pngs from your book if you want to use MATLAB or Seaborn you can do that
you can also include interactive outputs things that would normally output like HTML and JavaScript in the Jupiter
notebook so things like mocha or plotly or Altair those will also render just
fine in Superbook as well so you can use it to end that interactive content in your books okay so that being said I
want to quickly give a note of warning that there may be can some confusion
that I’m introducing here because as I mentioned before there’s a new version of Jupiter book that’s coming out quite
soon and I decided that I think it would be neat more interesting and potentially were youthful to talk about the new
version of Jupiter book as opposed to focusing on the newer version of Jupiter book that said the new version of
Jupiter book is still like very much so in an early beta kind of form so some things might change some things might be
a little bit different than if you just go to Jupiter book morgue but by and large the experience is pretty much the
same I think that the user facing workflow for Jupiter book shouldn’t change very much when it moves to new
build system would be hopefully a new faster more feature full back-end that’s
that’s working under the hood and I’ll go into a little bit of detail about that later so how do you create a
Jupiter book well in order to have in order to build a Jupiter book you basically need two things
o you need three things you need a collection of notebooks and markdown files that’s your book content you need
a table of contents file that just defines the structure of your book and then there’s an optional configuration
file that just allows you to you know turn on different kinds of features and things like that and so just as a little
helper Gepetto book comes bundled with a create function it basically just creates a
little lightweight template book that you can use – you know build on top of or you could
just use to see how about the structured so then you can adapt your own contents it’s a Jupiter book so if you type in at
the command line Jupiter book and create my book name I should say this is after you know pip installing Jupiter book once you run
Jupiter but create and then give your book a name it’ll generate a collection of files and a new folder for you and
those files are basically the three things that I just mentioned before so first off in your config yamo file it’s
just a collection of key-value pairs that define things like the title of your book who’s the author do you want
to show the sidebar by default or not you want to include binder links along with your book content there’s also a
table of contents file and this is the file that actually defines the structure of your book so the very first entry
here is going to be the first page the people land on you can define sub pages for any given page and then that allows
you to kind of nest sections within each cell and then files are just the path to
a given either I play a new file or a markdown file relative to your books
folder and then finally just have a collection of content files as I mentioned before we want that to either
be markdown files or Jupiter notebook files so I have a question for you Chris
relate up to this can you execute Jupiter book created sort of creates this cookie cutter template for you
that’s makes like a starting at the start of your book right when and then
it creates a table of contents for you how much is in that like is it just these exact files or does it create like
extra chapters and sort of subsections for you I will show you in a second oh
[Laughter]
okay so once you have like this set of content in a folder it’s really all you need to actually build a book so when
you run Jupiter book build and you point it to a folder that has that content inside of it what it does is it collects
all of those content files and then it renders them as a static HTML site that
you can then you know either deploy it online nullify or gh-pages whether you can just browse locally to see how things look
and importantly you want to make sure that you keep like all of the information that’s there in a notebook like the inputs the outputs eventually
you want to also include extra features for publishing like citations and cross-references and whatnot so after
you build your book you went from what was originally this collection of configuration and content files and now
we’ve created a new folder called build right now there will be one output in
that folder for HTML but in the future there’s going to be other kinds of outputs for things like you know PDF or epub or docx that kind of thing but
everything that’s inside of HTML is just a fully functioning HTML website ok so
after having shown you this but a bunch of static information I’m gonna give this a shot to actually show you how it
looks so I’ve just got a little Jupiter lab instance running here and I have an
empty folder and if I Jupiter Jupiter well JB is sort of a shorthand for
Jupiter but called do the whole thing create my cool book then it generates a
folder my cool book and if I look inside that folder I can see that I basically have exactly the files that I just
showed before there’s one extra file that I didn’t mention which is a big check file which I can show some some
functionality I’m not going to cover the exact structure of these because I already told you about it before but
this is the configuration file this is the table of contents file it’s just a light light a lightweight little one
that points to these files that are here and then the content that’s inside of these things is just like your regular
old content that you would expect to see in a notebook so here’s a regular
Jupiter notebook and whatnot so now if I go to if I change into my book folder
inside Jupiter book build and I was pointed to the current folder that I’m in then under the hood it’s actually
using Sphinx as I had mentioned before it collects all of those files it resolves some stuff about the content
that’s inside and then it generates this build folder right here so if I look inside of the build folder this is
a functioning static website and in fact I can now go to Chrome build HTML and
open up my index file and that will show the Jupiter book that I just built again
it’s very bare-bones and simple because it’s just there to sort of show you how things are meant to look just to browse
really quickly if I go to this content with notebooks file if you remember this is B this is the notebook that was used
under the hood for this file and this is how it renders on the actual page itself it looks good do you notice they don’t
have any outputs here because I haven’t run them in this page if I were to run everything here now I’ve got the outputs
there if I rerun Superbook build and
then refresh this page now I’ve got outputs okay so it’s dependent on
whether you execute the notebook went before you execute build what the outputs are gonna look like in the HTML
the rendered HTML pages it is currently that’s only because we haven’t built in
the execution feature yet like automatic
execution of notebooks that need it basically okay so another thing I noticed is on the example book that you
showed us on a little bit earlier there was a link to interact with binder and
also a download link and this when you just showed only has the download link
for the notebook on the HTML that you just showed us so does that this binder
only hook up once it is a rendered page on gh-pages or something like that or is
it elsewhere that one though so that’s actually a feature that already exists I just haven’t turned it on yet then it’s
a related project that we’re working on it’s kind of another one of the under the hood pieces but a lot of Matic those
binder links for you so that that’s there as well but what minigame is referred to is for any of the pages that
have a Jupiter notebook under the hood if you know the github repository where
those notebooks are stored then you can automatically inject a binder link notebooks and then you’ll be able to use
that to immediately send users to like a fully interactive version that’s running on a Jupiter server and the cloud would
binder yeah it’s nice of those integrates so easily together as well
yeah I mean I think that a lot of the a lot of the reasons for that is that you know one of the philosophies the design
philosophies of the Jupiter project is is to try to design fairly modularly and adopt like common standards and
protocols across the different pieces of the project so that you know on your hub which is a service that runs something
like my binder not Ward also has the ability for other things to hook into it programmatically so basically I think
you see I’ll just show you that in the current Jupiter but Cupid org has the
current version of it so here you see these interact buttons up here if I am on a page that’s got a notebook under
the hood I’ve configured this Jupiter book to say hey the the collection of
notebooks is this github repository and the notebooks is at this subfolder in there and so then Jupiter book knows how
to build up these interact links programmatically and just send me over to a binder instance if I were to click
on that so I’m not going to wait for the further the binders and spin up but you can kind of see how a reader can really
quickly like jump into this interactive version of it if they want to okay so
that’s like a very quick link here’s how you go from zero to a working Jupiter book in 50 seconds is should I pause for
questions there anything like that or do you want me to keep on going forward you can keep going we have questions a little bit later in the in okay thing
okay cool so in that case for the next piece of
this I want to talk about where the Jupiter book project is moving in the
future a lot of what I just described some parts of it were this like new build system that I did that I had
mentioned but a lot of it was just thick
I think we I think Christmas from
Verizon mm-hmm I think just have a chat so he comes back yeah hopefully
everything is okay now might be you know what I might be yeah I don’t know
exactly what happened but we’ll see what happens Oh Christy back we lost you for a second hi first you forgot it was like right
when you started talking about the executable book friend except your number that would be great so the exact
a book project is a sloan Foundation funded project it’s a collaboration between four different
python-based publishing projects so IAB is an integrative biology project at
Northern Arizona University headed by Greg capresso on the Left quant econ is
an open textbook for quantitative economics which is run out of Australian National University and that’s John Sturges key in the middle who’s the PI
on that project and then Krystal is recently graduated from Imperial College
London and he’s been running a project on publish which tries to bring in more traditional publishing features into the
Jupiter ecosystem and then obviously there’s the Jupiter book project coming out of UC Berkeley and so the goal of
this grant is to basically align the interests and the lines of stacks of all these different projects and try to have
like a single unified Python stack for publishing as opposed to each of us having our own separate pieces and so
this is the project that’s kind of driving a lot of the new developments it’s going to make its way into the next
version of Jupiter book you can find just like an NEHTA website for the project that EBP jupiter book or thanks
for executive old book project so really quickly i want to talk about some of the
interesting features that are coming up in the pretty near future probably the next like month or so in what I’m just
calling Jupiter book 2.0 so really quickly just to start off I wanted to
show like the build chain that I’m talking about here as I showed you before it was it was sort of going
straight from like static I plan and markdown files into a static website and that process is largely going to be
the same we’re hopefully a few extra pieces in the middle so first off we still want people to be able to write
their content either in Jupiter notebook files or in markdown files but then we
want to have a step where they can execute those files and potentially cash their outputs so like a separate cache
that Jupiter book knows how to use and then we want when we actually build the
site to be able to draw from that cache if needed so that we don’t need to reiax acute the notebooks every single time
that we build the site and that way we know that the outputs that are there are like the latest outputs that that should
be there but we don’t have to wait for execution every single time to be a little part and that’s actually important because projects like quant
econ I think they have many many many dozens of pages and it takes something like two and a half hours to run all of
the material for their whole book and so for some of these larger more complex projects we need to have some ability to
cache the outputs and check whether or not those outputs are stale before we build the book and then secondly once we
have all of these built books we then want to grab some more like rich publishing style information from them
so we want to look for things like cross references between pages equation references citations we want to pull out
all of that rich metadata it’s like mix up a proper multi-page document and then
be able to use all of that when we generate the final HTML website and then
finally once we have that in rich document model we want to have multiple kinds of documents so if you want a
static web played like what you think currently does you can do that if you want to have a PDF that looks really
nice and professional and maybe flows through la tech you can how about I think that people are working on some
some docx and some epub outputs as well so that’s kind of at a high level what
the new build process is going to look like and as I mentioned before I think
one of the goals of the reworking of the back end is to use a little bit more of
a modular architecture in the Jupiter and the Python ecosystem so currently the 1.0 version of jupiter book is
basically a sing repository on github it does a ton of different kinds of things right it does
like new book page mill conversion it stitches them all together into a website and it adds all of the bells and
whistles around like interactivity and binder links and things like that what we’d like to do now is build on top of
sphynx which is a really excellent documentation generator and Python ecosystem and basically what we’ve
discovered is that if you want to build like a high quality publication style output then a lot of the features that
you need for that are pretty similar to some of the features that you need for just building high quality technical documentation for your projects and so
our hope is that by using Sphinx we’ll be able to align our stack more closely with the broader Python ecosystem
so rather than largely relying on a tool in the Ruby stack which many of us don’t
know as well and we have less of an ability to contribute you know upstream improvements to we can use Fink’s we can
modularize a lot of the pieces of legibility does into Sphinx extensions that the broader Python community can
use on their own now we’ll get into some details about that in a second and hopefully make this project feed into
that kind of broader open Python ecosystem as well so a couple of things
that that brings to mind them one of them is so I mentioned Sphinx here and
for those of you who are familiar with Sphinx you might have thought sinks doesn’t use markdown speaks uses restructured text where’s Jupiter
notebooks use markdown and that is true so that was one of our concerns with using Sphinx is that many authors are
just not familiar what’s writing in restructured text and so what we decided to do was build a very lightweight
flavor of markdown that is a first-class citizen in the sphinx ecosystem we call
this something called a mist or markedly structured text and basically the goal
of mist is just take jupiter notebook markdown the same markdown that we all write in jupiter and then add a couple
of syntactic pieces to let you use two of the most powerful tools and the sphinx ecosystem with our rules and
directives so really quickly for those of you who aren’t familiar rules the
directives they’re kind of like functions but in markdown so roll or a directive can have a name it
can have arguments it can have keyword value option configuration and then it can also have content and this sends the
general structure of each of those things so directives are more pie align blocks here you have a directive name
you can have arguments you can configure it and then you can add your content in their roles are in line with the text
itself so you have a role name and then you just give it some kind of interpreted text and these are patterns
they’re quite common in distinct ecosystem that allow you to control different kinds of behavior in your
final built HTML website so an example of each of these things is on the Left
this is how you could define like a figure in Jupiter book you would say okay I’m using the figure directive I
want to point it to a PNG file and I want to give it a caption and a label and name and then later on in a role
there’s a ref role which allows you to make a cross-reference to another item so say this is on a different page I can
insert my label here and because I’ve defined it over here in this directive then Sphinx will know how to
automatically insert a URL link that cross-references this figure on whatever page is over here and then it’ll link
you back to the page where it’s linked probably yeah exactly right nice that’s really cool oh yeah that was a good file
that you were doing earlier on yeah and I can I can show that up in a little bit if I if we’ve got some time for that for
the last section so let me let me try showing you what this looks like that
so I’m gonna go back to my to my page here this is my little mini build site
and I’ll edit this markdown file so a couple of examples of rules and directives let’s say that I’m writing
some content and then I want to include an extra little piece of information that’s like a sidebar sort of like a side notes that I don’t want to break up
my the flow of the main text of the page but I want to give like a little contextual piece of information to the
side for that I can use the sidebar directive so if I do sidebar like that and then I give it a title my sidebar
title here’s my sidebar content and I close
off my directive then when I rebuilt the book and go to my markdown page assuming
that that my internet works then I have a little sidebar content down here and
so this is a way you could sort of control the look in the feel and the structure of your page just by using
these little kind of lightweight markup pieces I can show you when you see I
think I can show you a citation as well so for if I before that there’s this big tech file inside the demo book which
right now because I’m apparently a vain person I’ve populated with all of the papers for my PhD thesis so this is one
way to get my agent so now if I wanted to cite one of those pages I’m typing
some content and now a citation I can use the cite rule and I just pass it the
label that’s inside that FinTech file pretty cool easy
and now when I rebuild my book and
refresh this then I’m typing some content a citation this links out to the
page with the bibliography is inserted right here at the bottom extremely cool that’s so nice
nice Vaizey yeah so so that’s like two examples of how you can use this little
lightweight extra markup syntax to control new kinds of functionality in
your page and the other kind of cool thing about that is that again these are kind of like functions which means that
people can extend them in lots of different kinds of ways and actually the Sphinx ecosystem has a huge collection
of extensions and tools and pieces that are already out there for extending the functionality
of Sphinx itself and thus by extension jupiter book for example that that
citation syntax that I was using that uses a Sphinx extension called speak spin tech to allow textile references in
your documentation extremely nice I just
want to say is something goes to deal with citations that like makes it so much easier features to talk about if
there’s full-time we’ve got time with leak proof a few features yes yeah okay
cool okay so then a couple of extra things one of them is as a part of this process
again modularizing the stack that’s there we also needed the ability to have
like a first-class citizen for rendering notebooks in science Fink’s and that’s a project that we’re calling this and so
this is a project that builds on top of the sphinx sorry the mist parser that i mentioned before and this allows you to
basically include jupiter notebooks directly into your Sphinx bill so there’s straight from I plan D into your
instincts documentation and has all the nice pieces for you know including outputs and things like that and again
that’s a tool that but the the mist parser as well as miss Denby those are
both completely modular tools that don’t inherently know anything about Jupiter bugs so the tool that anybody can use in
the Sphinx ecosystem so either write their documentation purely in markdown or to write documentation with Jupiter
notebooks and included in their own open source projects another piece of this
project is the upstream contributions to a tool called Jupitus joopa text is a really
interesting two way conversion tool between jupiter notebooks and various text-based formats so there is now a
jupiter a Jupitus extension for nist markdown and what that means is that you
can go from an iamb file here and have lossless two-way synchronization between
this file and a markdown representation of that file so basically like code
cells become tick-tick-tick code cell and you get a kernel the one thing that it doesn’t transfer
over are the outputs because that would be pretty heinous to try to put into a markdown file but I’ll get to that in
just a second and then another kind of interesting tool that that we’ve just
been playing around with things once you have the ability to represent all of this information in Spanx like across
multiple pages of your book then you can start to do some pretty interesting things so like one pattern that we’ve seen is that a lot of people they run
their analyses in one notebook and they’ll have this kind of like negative complex a bunch of different you know a
bunch of Python or R code inside tons of different plots tons of statistics that they’re calculating and then in another
notebook or in another markdown file they’re actually writing the document itself and so we tried to sort of enable
that a little bit more with a little sub module that we called glue and what glue allows you to do is embed snippets of
information in a notebook and attach them to a key so that later on in your
book you can reference that key and insert its value into whatever other page you’re working from and so just a
really quick example of what I’m talking about there so in this figure here I generated this plot you know a bunch of
lines and attached to it is this map plot little figure object if I do from
mist and be import glue and then I say glue my cool figure and then pass it
Figg else that display is false so if display is true then it’ll actually try to display the figure when you glue it
just is like a sanity check what I’ve just done is I’ve taken this figure and
I’ve attached it to this key name inside the notebook and now well then let’s
let’s do another thing here so you have some good up here right so I’ve got data let’s just calculate the mean of my data
and I’ll say mean equals me and then
I’ll glue my mean to the variable I just created called
me so now I’ve glued two different keys
into my notebook my cool figure and my meet if I go back into this markdown
file here then I can do things like that
so use the glue directive and pass it
the key that I added in this notebook so my cool figure let’s just see what
happens there and let’s add some extra stuff to my cooler here and here’s a cool caption
Chris we just go to probably one more minute on this second sir yes and then
here is my statistic and then my mean
and now if I rebuilt the book okay if I
rebuild the book then glue normally but I’m gonna skip this piece because I
don’t think that we have time to go through it so anyway that kind of like gives you again this is all the very
alpha content and as I mentioned before this is sort of like where we’re trying to get to is increasing the ability to
sort of enrich the the relationships between the different pages that are
there in the various content pages and so the last thing I’ll just mention is
if you’re interested in trying any of this stuff out then what I recommend is checking out betas like the newer kind
of build interface for Jupiter book some of those modular components instincts that I mentioned the mist parser and
mist NBER both of those links and then if you just want to follow the project at large at the bottom of the page those
are executive old book projects as well awesome thanks for my flat crease that
was really enjoyable and I wish we had that when I was at university because I was just staring at boring lecture
slides and they were going through them but it’s great to see so that now that we’ve discussed about where the
project is heading so as a roadmap section we’re going to move on to the Q&A section soma Deccan do you want to
keep the first question off yeah okay so our first one is from Phillip abhi
follow-up statement to both your statement regarding the future interaction between Jupiter book and
Spinks notebooks are mostly a markdown based well Sphinx is mostly restructured
test I know that recently syncs allows markdown – isn’t this adding an additional layer of complexity and then
they have a clarification is the EBP going to be Jupiter book 2.0 are they
going to be two separate projects thank you yes so two the first one it definitely
is adding a layer of complexity in the sense that there is like I would say it’s more like cognitive complexity I’m
the users side because it’s another thing for them to think about the goal
of markdown we’ve all of adding markdowns of Sphinx was really trying to stick with the most common version of
markdown that’s out there which is called common mark which is closer to a community standard than anything else
that exists in markdown ecosystem and then just adding like a few extra pieces that are needed to get it working
instincts one of the challenges that we ran into was we wanted to utilize the
Sphinx ecosystem but we found that for a lot of people writing and restructured text was just like a non-starter
personally I read in restructure text a lot and I enjoy it but especially for a
lot of our introductory users again a lot of people that are using this stack or like you know professors who are not
super comfortable comfortable with like the deep guts of the Python ecosystem but they’re very familiar reading
markdown because they use github and things like that so really our goal was a few lightweight pieces on the Suites
ecosystem to allow people to write a documentation purely in markdown and
really like Sphinx itself is meant to be extensible in this way so I think we’ve
lost Chris again we can pause for a minute and hopefully he’ll come back um but you know while we’re waiting if
any of you have any more questions please add them to the thing we’ve just
lost Chris he’ll be back I’m sure he’ll be back this is gonna be good but thank you for your precision so far
everybody this has been really great it’s been an old episode it’s very
enjoyable no curse copy and chris is back yeah Chris you’re back so so I
think I think that where I was caught up was like in many ways Fink’s is meant to be expensable extensible in this way so
it’s just swings builds in a lot of different hooks so that you can add new kinds of functionality restructuredtext
is the markup language that they went with by default but they very intentionally wanted it to be possible for people to write and other kinds of
markup languages and so that’s what the mist parser is a parser for sphinx that’s an extension of a sphinx
ecosystem for markdown so to answer your second question basically the executive
Oh book project is like a meta project under which it’s basically a collaboration that is explicitly attached to this grant that project will
contribute to and build a number of tools in the open-source Python ecosystem miss 10b miss parser Sphinx
but also just up streaming a lot of a lot of pieces – like Cupid text the Jupiter Yuka system and whatnot Jupiter
book will I think be a more kind of controller process over these different pieces and I think that eventually the
executive book project will dissolve because the grant will run out and at some point in the next you know six
months or so the grant goes for another three years but at some point of the next six months or so we’re gonna pivot
more of our effort towards like building out the open-source communities around these projects up streaming as many
contributions as we can to other projects and the Jupiter and the Python Sphinx ecosystem and you know trying to
build a more proper like sustainable open-source community around oh so super nice yeah question in the
question box from Sophia yang can I use Jupitus with Jupiter book so that my
content files would be dot PI files instead of ipython notebook bios yes yeah it’s not in there yet in the new
version but eventually yeah but we want this for anything that is like a valid joopa text input should also work with
Jupiter book I think that the missed markdown flavor of Jupitus will be
treated in some kind of special way probably for the purposes of caching the outputs and storing them for later on so
that you know when you basically when you rebuild a book it’ll first check and say okay which pages have you edited and in those pages
did you edit any of her code if not then I don’t need to rerun all the code I could just use the cached outputs but if
so I need to re execute it I think some of that logic might be like Jupiter notebook or NIST notebook specific but
in general yeah we want to support anything that you put textual support awesome great thank you for that and
thank you very much for your questions everyone listening right so now it’s time for our famous rant and rave
section where we both get 15 seconds to either rat or rave about what ever topic
we want so Chris it’s your soapbox yeah so I think that for me the thing that I
wanted to rave about is I guess an offense also related to Kairos but for
example I think one of the things that I really enjoy about this is that you get
a different kind of perspective into people’s lives than you normally do when you’re just meeting them in conference
rooms and whatnot and one of the things that I particularly enjoy is watching the number of dogs cats and children
that decide to come in and like pawing at their their parents faces in the middle of zoom chats and Skype
meetings and whatnot I don’t know why but I think that I find it like a nice kind of way to personalize what is a
difficult situation for a lot of people and working from home not seeing a lot of other people and it brings me a
little bit of happiness to get those little interruptions whether they’re there once it or not on the other end I
enjoy them um okay well I’d like to rave
about libraries and librarians I think right now we all are thinking about ways to get the content we want while we’re
at home and there’s a lot of really great resources going around that libraries offer that are not in person
so you don’t just have to get books in person there’s lots of really good apps where you can download books but you can
also download things like comic books or audio books and libraries are like collecting a lot of resources related to
that so I’d like to thank all of our librarians they’re amazing and thank you for being great thank you very much
chris is sort of in line with you but I have a bit of a rant and it’s more a hypothesis now that everyone’s working
from home I just look at snapchat I look at social media and everyone seems to be more healthy and really enjoying working
from home so I just wanted to rant by the fact that I think there’s gonna be a bit of an uprising in a lot of companies
where people are going to demand now that they work from home so that’s my little rant but yeah thank you very much
that’s all we have time for today thanks for watching and thanks for listening you can find us on twitter at at
and at con site AI so if you’re interested in also funding open-source
projects including jupiter book you can find all the project roadmaps at Quonset comm slash projects chris so where can
people find you in jupiter book yeah so me personally and just see whole graph on twitter as well as on github I spend
a lot of my time there as well as a veggie burger a community forum that discourse that Jupiter or not orgy the
Jupiter book project I shared a couple of links before and could include those and you know whatever materials we
follow up with on this webcast but Jupiter book that work for the current version of Jupiter book beta Jupiter
book work for the the new version of it and then github.com / executive o book
project is just the place where we are like managing the process of the project
itself and you can kind of keep up to date with some of the movement there if you’re interested in following along or
contributing as well great thank you Chris thank you for your time and thanks everyone for listening join us again
episode 4 a pointed discussion on Apache arrow alright thank you very much
[Music]
