Earlier this month, Python rose to number one in the TIOBE Index, marking the first time in 20 years that a language named C or Java was not at the top of the list. It’s a nice feather in the Python cap and marks the pinnacle of language reach. But what is behind Python’s impressive success?

Python started out fairly modestly, as a hobby to occupy Guido van Rossum during the holidays of 1989. Little did he know he was launching a programming language that would have a lasting impact on a segment of the computing world for over 30 years. .

Van Rossum created Python to be a general purpose interpreted programming language. His initial design goals for Python were that it would be an easy and intuitive language to learn, as understandable as plain English, suitable for everyday tasks, and open source.

The legacy of these design goals is clearly evident when speaking with members of the Python community, which is a growing group of individuals who use Python for a wide variety of things. While programmers ostensibly write Python for work, it feels like they really enjoy it.

“I chose Python because it was a fun language to code,” says Calvin Hendryx-Parker, CTO and co-founder of Six feet in the air, a Fishers, Indiana-based consulting firm that does a lot of custom Python coding.

“I like it because it’s not just a rough job,” he continues. “Everything is very innovative and creative. It is more of an artisanal know-how. Engineers build the same thing day in and day out, but artisans have to solve problems creatively because the environment around them is constantly changing.

Six Feet Up was founded to build custom web applications, but customers in the Midwest are increasingly turning to the company for help getting big data projects into production. While Python isn’t the only language involved in these projects, they usually involve a fair amount of everyone’s favorite scripting language.

“The good thing is that a lot of big data tools are starting to be built in Python,” says Hendryx-Parker. “It’s that kind of lingua franca for all things data.”

The massive Python ecosystem

The Python data ecosystem is large and growing. From Python wrappers for Keras and TensorFlow to machine learning packages like scikit-learn and NumPy, from Python interfaces for Spark and Dask clusters to experimenting with code in a Jupyter or Pandas notebook, data scientists and engineers can do pretty much everything they need without leaving Python’s comfortable syntax.

Python’s flexibility is at the forefront for Prefect, a Washington, DC-based company that develops a Python-based data pipeline tool. According to CTO Reeve Chris White, Python is the perfect tool for the job.

“At a high level, our design sensitivity is sensitive but deeply configurable defaults, so you can kind of get something to work really fast if you really want to, but when you really want to get in the weeds, you can do that too. ” White says. “Python really helps us achieve this goal, because at the end of the day, Python is a very flexible language. You can write a script. You don’t need to know what it means to compile or anything. You’re just writing a script.

It all goes back to van Rossum’s design ethic for Python, which was simplicity rather than complexity. The barrier to entry for Python is incredibly low, thanks to its imminent readability. This allows Prefect to use Python with its junior coders and more experienced programmers.

“It’s one of the few languages, at least to my knowledge, that can be used as a raw scripting language. You just run it. You don’t have to know anything else, ”White says. “So it’s very accessible. and of course you can wrap things up. You can write C extensions and things like that.

Flexibility and performance

Speaking of C, performance isn’t one of Python’s strong points. This is one of the tradeoffs for imminent flexibility and accessibility. Through C extensions, there is a way to achieve higher performance. But in many cases, this performance is not necessary.

“There are of course other programming languages ​​that will be much faster than Python in the most extreme extreme cases, for real-time processing or something like that,” White said. “But in 90% of the data that people actually process… you don’t need that speed. “

Another historical negative in the Python directory is type security. Python is a strongly typed language, which has drawn criticism from some. However, people have started to realize that this is not as negative as they might have thought, White says.

“A lot of the things that people initially complained about, like the guy’s speed and safety, people start to relax a bit. Maybe they’re not that important to some of the applications people are dealing with, ”he says. “That’s the bottom line – you can’t do a lot of type checking with the data. Most of the data is so unstructured. It’s really hard to imagine how you would structure this. But Python at least allows you to start iterating and understanding very easily, whereas other languages ​​would kind of force you to adopt a scheme very early on.

Development speed is a big plus in Python’s corner. Because the language is so accessible, developers can quickly put something together without making it a major project. The casualness of Python coding may be at its peak in Juypter, where you can run Python in place in the browser, even if things aren’t working.

A force multiplier

When placed in the right hands, Python tools can also function as a force multiplier, delivering sophisticated engineering and data science capabilities that go beyond what would be possible if we were all to be experts in C or in Java. White relays this story from a Prefect client who used the Prefect tool (a Python creation) to orchestrate fairly complex data workflows.

“The customer said,“ I’m not really a software engineer by training, but I just wrote what I think is a very robust, fault-tolerant batch workflow and deployed it to Kubernetes, ”White says. “He said ‘I don’t care what those words mean, but the prefect gave me the tools to make it happen.'”

While Python lowers the barrier to entry, thanks to Python’s flexibility and enthusiastic user community, one can also write perfectly manicured, high-quality code that will be an absolute blast for your friendly DevOps engineer to manage.

“Python allows you to build something very quickly,” says Hendryx-Parker. “But it also lends itself to easily converting them into a well-designed software project with all the cool unit testing features that can be deployed with CI / CD – all the kind of modern trim that surrounds it. “

The Open Source community

Another advantage in favor of Python is the open source community that has built up around it, according to Hendryx-Parker, who has participated in Python conferences for almost 20 years and who is the founder of the local Python user group in Indianapolis. , IndyPy

“It’s a bunch of amazing people who want to make sure they’re welcoming and open to other amazing people to come and be a part of this community, which isn’t always easy in a host of technologies,” he said.

There are thousands of add-ons, add-ons, and integrations for Python, most of which are open source, but some are poorly maintained. This can sometimes make it difficult to avoid Python packages that have succumbed to old age or rot, says Hendryx-Parker.

On the flip side, when one comes across a useful but overlooked library, it also gives Python users the opportunity to step up and give back to the Python community by volunteering to help maintain the code, says- he.

“We encourage our customers who are in this situation to spend time in their budget to improve these open source projects,” says Hendryx-Parker. “I think it’s good all around. It’s a win-win-win. If they can get that budget and they know they need to have that capacity in their supply chain, it’s not a given.

Compared to the well-documented ecosystem that was built around R, the Python ecosystem is a bit less orderly, which has its pros and cons, White says.

“R really got a lot of traction because it looks like a standard. It has the same universe of packages that work really well together, and so I think that pushes people to R, ”he says. “Python is a bit like the Wild West. You can kind of do anything. There is no set of packages as strictly standardized as the Tidyverse is in R.

On the other hand, the Python community is so large and open that users have many sources of help. “There are a lot of great documents out there, so you can always get the answers you need very quickly,” White explains. “But we have to go out and watch. It is not as autonomous.

The future of Python is bright. It’s at or near the top of everyone’s skills list for data science and data engineering. With its current momentum, Python seems destined to keep growing until something manages to dislodge it from the top of the heap.

Related articles:

Three reasons why Python is the AI ​​Lingua Franca

Python attracts rookies as well as pros

Is Python strangling R to death?