Planet Python

Last update: November 09, 2020 01:47 AM UTC

November 08, 2020

Weekly Python StackOverflow Report

(ccxlix) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2020-11-07 18:30:25 GMT

1. Dataframe removes duplicate when certain values ​are reached - [7/2]
2. What's the Python equivalent of Julia's `@edit` macro? - [7/1]
3. Longest path finding with condition - [6/2]
4. How to check if a column contains list - [6/2]
5. Solve equation with sum and index using Sympy - [6/1]
6. Python packaging in 2020 - [6/0]
7. How to split string at specific character and build different string combinations - [5/5]
8. Best way to apply function with many iterations to pandas series - [5/3]
9. How can I make a distance matrix with own metric using no loop? - [5/2]
10. Looping and finding composite/prime numbers and saving it to a set - [5/2]

November 08, 2020 01:04 PM UTC

November 07, 2020

Anwesha Das

Using Mailvelope with Yubikey in Linux

Mailvelope is an extension on web browsers to send end to end encrypted emails. This is a good option available to the users to send end to end encrypted without changing the email service they use. It is licensed under AGPL v3, making it Free and Open Source software. The code is there in Github for the community to have a look. This can be added as an extension to the - Chrome, Firefox and Edge browsers to securely encrypt emails with PGP using your email providers.
​
Mailvelope does provide end to end encryption for the email content but does not protect the metadata (subject, IP address of the sender) from third parties. As most of the email encryption tools, it does not work on the mobile browser. There is a detailed user guide on Mailvelope from the Freedom of the Press Foundation, which is really helpful for the new users.
​
By default, Mailvelope uses its own keyring. To use my Yubikey along with GnuPG keyring, I had to take the following steps:
​

Install gpgme

​
We need gpgme installed. On my Fedora 33 I did
​

$ sudo dnf install gpgme -y

​

For Chrome browser

​
​
We have to create gpgmejson.json .json file in the ~/.config/google-chrome/NativeMessagingHosts directory write the following json in there.

{
    "name": "gpgmejson",
    "description": "Integration with GnuPG",
    "path": "/usr/bin/gpgme-json",
    "type": "stdio",
    "allowed_origins": [
        "chrome-extension://kajibbejlbohfaggdiogboambcijhkke/"
    ]
}

​

For Firefox

​

mkdir -p ~/.mozilla/native-messaging-hosts
​

After creating the native-messaging-hosts directory inside the Mozilla directory, add gpgmejson.json file there with the following content.
​

vim ~/.mozilla/native-messaging-hosts/gpgmejson.json
​

​

{
    "name": "gpgmejson",
    "description": "Integration with GnuPG",
    "path": "/usr/bin/gpgme-json",
    "type": "stdio",
    "allowed_extensions": [
        "jid1-AQqSMBYb0a8ADg@jetpack"
    ]
}
​

​
Remember to restart the respective browser after you add the .json file. Then go to the Mailvelope extension to select the GnuPG keyring.
​

November 07, 2020 04:01 PM UTC

November 06, 2020

ListenData

Translating Web Page while Scraping

Suppose you need to scrape data from a website after translating the web page in R and Python. In google chrome, there is an option (or functionality) to translate any foreign language. If you are an english speaker and don't know any other foreign language and you want to extract data from the website which does not have option to convert language to English, this article would help you how to perform translation of a webpage.

What is Selenium?

Translate Page in Web Scraping in R and Python

November 06, 2020 02:23 PM UTC

PyCharm

PyCharm 2020.3 EAP #4: Pair Programming, Intelligent Text Proofreading, and More

The fourth build of PyCharm 2020.3 is now available in the Early Access Program.

This week’s EAP is devoted to platform-wide features, now bundled in PyCharm: Code With Me, a tool for collaborative development and pair programming, and Grazie, an intelligent text proofreading assistant. For the full list of issues fixed in this update, see the release notes.

We invite you to join our EAP to try out the latest features we have coming up, test that they work properly in your environments, and help us make a better PyCharm for everyone!

DOWNLOAD PYCHARM 2020.3 EAP

Highlights

Code With Me

PyCharm EAP 2020.3 now includes Code with Me features by default. If you are practicing pair programming or teaching Python online, it is definitely a tool you will fall in love with.

As a host you can define the level of permissions you want to grant to your guest users. You can allow others to edit files, access the terminal, and run code in your project. Alternatively, you can limit their access level to only viewing what happens while you are live coding.

Apart from the ability to develop collaboratively, a notable feature of Code With Me is its support of code insight features on the guest side, such as code navigation and autocompletion.

Learn more about Code With Me features in this blog post.

Grazie

Grazie provides intelligent spelling and grammar checks for text that you write in PyCharm. It recognizes natural language constructs in programming languages (Python, Java, and others), markup languages (Latex, Markdown, XML, and HTML), comments, commit messages, and more.

Grazie can help you check grammar in your text files (for example in README.md). To get the most out of it, you can also configure it to check textual elements in your source code such as string literals, comments, or commit messages.

Grazie is set to check English text by default, and you can enable over 15 other languages, including German, Russian, Chinese, and others, by opening Settings/Preferences and selecting Editor > Proofreading.

NOTE: All verification is performed locally, inside your IDE.

Learn more about Grazie’s functionality in the PyCharm Web Help.

Web development

Tailwind CSS plugin

PyCharm 2020.3 now supports the Tailwind CSS plugin. Enjoy autocompletion of Tailwind classes in your HTML files and after the @apply directive. You can make customizations using tailwind.config.js files and get autocompletion for them too.

All you need to do is install the plugin from Settings/Preferences|Plugins|Marketplace > Tailwind CSS.

Notable fixes

• Plots layouts in SciView are now displayed correctly.
• Fixed issue with types and parameters not being shown in code completion popup inside the Python Console.
• We’ve implemented easy navigation through the error path within a project for webpack.

Ready to join the EAP?

Some ground rules

• EAP builds are free to use and expire 30 days after the build date.
• You can install an EAP build side by side with your stable PyCharm version.
• These builds are not fully tested and can be unstable.
• Your feedback is always welcome. Please use our issue tracker and make sure to mention your build version

How to download

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP. If you’re on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP and stay up to date. You can find the installation instructions on our website.

This is all for today! For the full list of features and fixes present in this build, see our release notes. We also encourage you to stay tuned for more improvements, so come and share your feedback in the comments below, on Twitter, or via our issue tracker.

The PyCharm team

November 06, 2020 12:24 PM UTC

Real Python

The Real Python Podcast – Episode #34: The Python Modulo Operator & Managing Data With SQLite and SQLAlchemy

Are you ready to move beyond flat files for your data in Python? Maybe you're not sure where to start with databases and SQL. This week on the show, David Amos returns with another batch of PyCoder’s Weekly articles and projects. We cover a Real Python article about managing data with SQLite and SQLAlchemy.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 06, 2020 12:00 PM UTC

Codementor

Guide To Frontend Testing Using Django

Front-end testing should always be incorporated in the workflows. This guide will teach you the basics of front end testing using Django.

November 06, 2020 07:08 AM UTC

[Part 2/2] Persist data to the Ethereum blockchain using Python, Truffle, and Ganache

The previous post (https://dev.to/gcrsaldanha/deploy-a-smart-contract-on-ethereum-with-python-truffle-and-web3py-5on) demonstrated how to write a simple smart contract with Solidity and deploy it...

November 06, 2020 01:58 AM UTC

[Part 1/2] Deploy a Smart Contract on Ethereum with Python, Truffle and web3py

In this tutorial, we'll write a simple smart contract, deploy it to a personal Ethereum blockchain, and call the contract from a Python script.

November 06, 2020 01:58 AM UTC

Using Python Generators to avoid extra service calls

How to use Python generators to reduce the number of calls to other services.

November 06, 2020 01:54 AM UTC

November 05, 2020

Dataquest

Beginner Python Tutorial: Analyze Your Personal Netflix Data

Click the red button to submit your data download request.

When Netflix says it may take a month to get your data.

A sample of what the data looks like as a spreadsheet.

import pandas as pd

df = pd.read_csv('ViewingActivity.csv')

df.shape

(27354, 10)

df.head(1)

df = df.drop(['Profile Name', 'Attributes', 'Supplemental Video Type', 'Device Type', 'Bookmark', 'Latest Bookmark', 'Country'], axis=1)
df.head(1)

df.dtypes

Start Time    object
Duration      object
Title         object
dtype: object

df['Start Time'] = pd.to_datetime(df['Start Time'], utc=True)
df.dtypes

Start Time    datetime64[ns, UTC]
Duration                   object
Title                      object
dtype: object

# change the Start Time column into the dataframe's index
df = df.set_index('Start Time')

# convert from UTC timezone to eastern time
df.index = df.index.tz_convert('US/Eastern')

# reset the index so that Start Time becomes a column again
df = df.reset_index()

#double-check that it worked
df.head(1)

df['Duration'] = pd.to_timedelta(df['Duration'])
df.dtypes

Start Time    datetime64[ns, US/Eastern]
Duration                 timedelta64[ns]
Title                             object
dtype: object

# create a new dataframe called office that that takes from df
# only the rows in which the Title column contains 'The Office (U.S.)'
office = df[df['Title'].str.contains('The Office (U.S.)', regex=False)]

office.shape

(5479, 3)

office = office[(office['Duration'] > '0 days 00:01:00')]
office.shape

(5005, 3)

When you realize how much time you've spent watching the same show.

office['Duration'].sum()

Timedelta('58 days 14:03:33')

office['weekday'] = office['Start Time'].dt.weekday
office['hour'] = office['Start Time'].dt.hour

# check to make sure the columns were added correctly
office.head(1)

%matplotlib inline
import matplotlib

# set our categorical and define the order so the days are plotted Monday-Sunday
office['weekday'] = pd.Categorical(office['weekday'], categories=
    [0,1,2,3,4,5,6],
    ordered=True)

# create office_by_day and count the rows for each weekday, assigning the result to that variable
office_by_day = office['weekday'].value_counts()

# sort the index using our categorical, so that Monday (0) is first, Tuesday (1) is second, etc.
office_by_day = office_by_day.sort_index()

# optional: update the font size to make it a bit larger and easier to read
matplotlib.rcParams.update({'font.size': 22})

# plot office_by_day as a bar chart with the listed size and title
office_by_day.plot(kind='bar', figsize=(20,10), title='Office Episodes Watched by Day')

The Office views by day, Mon-Sun.

# set our categorical and define the order so the hours are plotted 0-23
office['hour'] = pd.Categorical(office['hour'], categories=
    [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23],
    ordered=True)

# create office_by_hour and count the rows for each hour, assigning the result to that variable
office_by_hour = office['hour'].value_counts()

# sort the index using our categorical, so that midnight (0) is first, 1 a.m. (1) is second, etc.
office_by_hour = office_by_hour.sort_index()

# plot office_by_hour as a bar chart with the listed size and title
office_by_hour.plot(kind='bar', figsize=(20,10), title='Office Episodes Watched by Hour')

The Office views by hour, AM-PM

When you realize your Netflix viewing habits have led to you finishing a cool project.

Charlie Custer

Charlie is a student of data science, and also a content marketer at Dataquest. In his free time, he’s learning to mountain bike and making videos about it.

The post Beginner Python Tutorial: Analyze Your Personal Netflix Data appeared first on Dataquest.

November 05, 2020 10:50 PM UTC

PyCharm

Webinar: “What It’s Like To Do Instructional Videos on YouTube” with Nelson Jamal

Want to learn about producing technical videos on YouTube and gaining an audience? Nelson Jamal, who has experience with both, joins us to take a deep look at his soon-to-be-released PyCharm tutorial on YouTube: planning, recording, and publishing. He’ll compare these with some of his other popular videos and what he’s learned along the way.

• Thursday, November 19
• 12:00 Eastern Standard Time
• Register here

Speaking To You

Nelson is a Software Engineer and Entrepreneur. He provides easy to follow programming courses on Amigoscode and has a YouTube channel where he teaches and inspires people around the world how to code. You can also find him on Discord.

Nelson’s main goal is to encourage and inspire people from all ages to experience the career path of Software Development. He has a strong belief in programming and wants to make it accessible to anyone by providing intuitive and easy to learn programming courses and tutorials allowing students to get ready for job opportunities, university and bootcamps.

November 05, 2020 09:13 PM UTC

Codementor

How to Learn Programming Online: Best Websites to Start With

In a constantly changing world, more and more new professions appear and gain popularity. But there is one of a few industries that remains popular decades later. It is programming.

November 05, 2020 04:18 PM UTC

Test and Code

137: Become an Author - Matt Harrison interviews Brian Okken

Matt Harrison, author of many Python books, is putting together a course, Effective Book Authoring, to help other people write and publish books.
As part of this course, he's including interviews with people who have already written books, including me. This is that interview.

We discuss:

• Why I wrote "Python Testing with pytest"
• Self publishing vs working with a publisher
• The writing, editing, and publishing process
• Writing format
• Book promotion
• Advice to other writers

Special Guest: Matt Harrison.

Support Test & Code : Python Testing for Software Engineering

Links:

• Effective Book Authoring — Matt's course
• Python Testing with pytest - Simple, Rapid, Effective, and Scalable

November 05, 2020 04:00 PM UTC

Python Morsels

Python Doesn't Have Type Coercion

Transcript:

Many programming languages have something called type coercion; it's where the language will implicitly convert one object to another type of object in certain circumstances.

Python does not have type coercion.

Numeric Types and Coercion

If we add together an integer (2) and a floating-point number (3.5) in Python, we'll get back a floating-point number:

>>> x = 2
>>> y = 3.5
>>> x + y
5.5

Python did not coerce the integer into a floating-point number; we don't have type coercion in Python. Instead, Python delegated to the integer and floating point numbers and asked those objects to add themselves together.

Whenever Python sees x + y, it calls the __add__ method on x passing y to it:

>>> x.__add__(y)
NotImplemented

In this case Python got NotImplemented back because integers don't know how to add themselves to floating-point numbers. This special NotImplemented value was returned by the__add__ method of the integer object to let Python know that x (an int) doesn't know how to support the + operator with y (a float).

When Python sees this special NotImplemented value, it then attempts to ask y whether it knows how to add itself to x. To do this Python call the __radd__ method on y, passing it x:

>>> y.__radd__(x)
5.5

This adds the floating-point number to the integer from the right-hand side of the plus sign (r is for "right" in __radd__) and returns 5.5.

So no type coercion was done here, instead, one of these types of objects knows how to operate with the other type of object when used with the plus operator.

Python Objects Don't Support Type Coercion

A counterexample of this is strings.

What happens if we try to use the + operator between a string and a number in Python?

>>> name = "Trey"
>>> x
2
>>> name + x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to str

Many programming languages would make the string Trey2 above: they would concatenate that string and that number, by coercing the number into the string. In Python, we get an error instead.

The reason is that strings in Python don't know how to use the plus operator with numbers and numbers in Python don't know how to use the plus operator with strings, which means our code doesn't work.

So to actually accomplish what we're looking for, we need to explicitly convert the number to a string:

>>> name + str(x)
'Trey2'

We've made a new string out of that number 2, and we're concatenating it to our string name to get another string.

Summary

Python don't have type coercion. Python doesn't ever implicitly converts one object to another type of object.

You'll always either rely on at least one of the objects you're working with knowing how to operate with the other type of object or you'll have to explicitly convert one of your objects to another type of object.

November 05, 2020 04:00 PM UTC

Python⇒Speed

Poetry vs. Docker caching: Fight!

Docker packaging is an exercise in shoving square pegs into round holes, over and over and over again.

Consider the Poetry packaging tool for Python. One of Poetry’s features can make Docker rebuilds slower, by breaking Docker’s caching.

And it’s not a bad feature, there’s nothing really wrong with it, it just—doesn’t fit.

Let’s see what the problem is, go over some workarounds—which have their own problems, obviously—and then briefly consider why everything about Docker packaging is always slightly broken.

Note: Outside the very specific topic under discussion, the Dockerfiles in this article are not examples of best practices, since the added complexity would obscure the main point of the article.

To ensure you’re writing secure, correct, fast Dockerfiles, consider my Quickstart guide, which includes a packaging process and 60+ best practices.

November 05, 2020 12:00 AM UTC

Spyder IDE

New features in Spyder 4's new debugger!

IPython is a great improvement over the standard Python interpreter, bringing many enhancements such as autocompletion and "magic" commands. When debugging, however, many of these features become inaccessible. With Spyder, we aim to bring back these capabilities and more for a truly premium debugging experience! (And believe me, I use this debugger a lot, and not only because I write code that might contain bugs :p).

In this post, I will describe the debugger improvements we've already made in Spyder 4, as well as those that are already implemented or under review for Spyder 4.2 and beyond.

Make the debugger more like IPython

IPython improves on the stock Python interpreter by adding syntax highlighting, completion, and history. We have done the same for the debugger!

The output is prettier (and easier to read) than plain black text, as it was in Spyder 3!

Code completion and history for the debugger use the same functionality as the IPython console, so you should not notice any difference in behaviour. Just press Tab to trigger autocomplete suggestions, and use the up and down arrows to recall previous commands.

This can result useful if you don't know every library's documentation by heart!

Another improvement that we added is the ability to write multiline statements.

This is better than getting SyntaxError: unexpected EOF while parsing, as was the case in Spyder 3.

Finally, IPython has a lot of nice magic functions you can take advantage of. I personally often use %matplotlib to switch between the qt5 and inline plotting backends, and %timeit to check how fast a given snippet of code is. Now, these can be used in the debugger as well!

Core improvements

When debugging code, a lot of time can be lost stepping through other Python packages. To avoid that, enable the new Ignore Python libraries while debugging option in Spyder's preferences under IPython Console > Debugger > Debug. This will skip all the built-in and third-party Python modules and help you focus on your own code.

Another pain point for me is the mix of Pdb commands and Python statements when using the interactive prompt. While debugging a script with a variable named step, for example, typing step in the Pdb prompt will not display the value of step, but will instead step through the code. To avoid this, the Spyder 4.2 debugger will give precedence to Python expressions over Pdb commands; prefixing an expression with an exclamation mark will explicitly run it as a Pdb command. This matches how the IPython Console handles clashes between magic functions and Python expressions, and contrasts with the previous behavior, where Pdb commands took precedence. You can toggle this under IPython Console > Debugger > Debug in Spyder's preferences.

Here, you can see I can have a variable named step, and use it just as I would in the IPython console!

While debugging, the context changes constantly, which can be annoying if you need to use a certain module in each frame. To address this, Spyder 4.2 will allow you to execute a snippet of code before any Pdb command. I personally use import matplotlib.pyplot as plt so I can use Matplotlib anywhere with no extra commands! You'll be able to set this up for yourself under Preferences > IPython Console > Debugger > Run code while debugging.

Matplotlib support

Speaking of using Matplotlib, two improvements will make your plotting life much easier while in the debugger. First, if you use the inline backend, the Process execute events while debugging option in Preferences > IPython console > Debugger (on by defaults) enables plotting after each command.

If you would rather use an interactive backend, Spyder 4.2 will allow you to do that too! The event loop will be running while waiting for a Pdb input, which will bring Matplotlib figures to life!

For example, interactive figures are great to zoom and pan.

Spyder Editor integration

By clicking next to the line number in the Editor, you can set a breakpoint to stop the debugger upon reaching that line. With a Shift-click, you can set a conditional breakpoint that will only pause execution if an expression you write evaluates to True. This is very useful if you are debugging a loop and want to go to a specific iteration, for example! While debugging, the Editor will show you the line on which execution is stopped. If you move around in the Editor, you can always go back to the current line by using the Pdb (w)here command.

Here, you can see the Spyder debugger is stopped at the breakpoint on line 385, which is highlighted. The next line has a conditional breakpoint.

Other Spyder panes

Several other Spyder panes are useful while debugging. Breakpoints gives you a list of all the breakpoints you have set throughout your code.

The Variable Explorer pane allows you to browse and modify the local namespace wherever the debugger is stopped.

The future

I'm excited for the new Frames Explorer pane, which is currently almost ready for Spyder 5. Usually while debugging, it can be hard to tell from where the current function was invoked, and one must often step up a long chain of calls to figure it out. With the Frames Explorer, you have direct access to all the relevant call information, and you can jump around the stack easily!

I can see where the current function was called from!

The next challenge I will be tackling is to enable the debugger to stop in threads. This is a bit difficult to implement because if two threads reach a breakpoint at the same time, only one of them can take control of the shell, leading to skipped breakpoints. This might be okay, but I'm still figuring that out.

Try out the new improvements in the Spyder debugger and let us know if they make your debugging easier!

November 05, 2020 12:00 AM UTC

Matt Layman

Create A Form Template - Building SaaS #78

In this episode, I created a template for one of my new forms on the new social media app that I’m building. We talked about context data, template styling, and special considerations for forms in templates. I had an empty template for the invite sending form to begin. I filled in a first attempt at the template with a header and displaying form errors. While building that, I added some context information that was needed for the display.

November 05, 2020 12:00 AM UTC

November 04, 2020

The Digital Cat

Public key cryptography: SSL certificates

In the context of public key cryptography, certificates are a way to prove the identity of the owner of a public key.

While public key cryptography allows us to communicate securely through an insecure network, it leaves the problem of identity untouched. Once we established an encrypted communication we can be sure that the data we send and receive cannot be read or tampered with by third parties. But how can we be sure that the entity on the other side of the communication channel, with which we initiated the communication, is what it claims to be?

In other words, the messages cannot be read or modified by malicious third-parties, but what if we established communication with a malicious actor in the first place? Such a situation can arise during a man-in-the-middle attack, where the low-level network communication is hijacked by a malicious actor who pretends to be the desired recipient of the communication.

In the context of the Internet, and in particular of the World Wide Web, the main concern is that the server that provides services we log into (think of every service that has your personal or financial data like you bank, Google, Facebook, Netflix, etc.) is run by the company that we trust and not by an attacker who wants to steal our data.

In this post I will try to clarify the main components of the certificates system and to explain the meaning of the major acronyms and names that you might hear when you deal with this part of web development.

Clarification: SSL vs TLS¶

In the world of web development and infrastructure management, we normally speak of SSL protocol and of SSL certificates, but it has to be noted that SSL (Secure Sockets Layer) is the name of a deprecated protocol. The current implementation of the protocol used to secure web applications is TLS (Transport Layer Security).

The story of SSL and TLS is rich of events and spans 25 years since its inception by Taher Elgamal at Netscape. In short, SSL had 3 major versions (the first of which was never publicly used), and was replaced by TLS in 1999. TLS itself has gone through 3 revisions at the time of writing, TLS 1.3 being the latest version available.

The TLS/SSL nomenclature is one of many sources of confusion in the complicated world of security and applied cryptography. In this article I will use only the acronym TLS, but I went for SSL in the title because I wanted the subject matter to be recognisable also by developers that are not much into security and cryptography.

X.509 certificates¶

While the problem of the identity in an insecure network can be solved in several ways, the solution embraced to secure the World Wide Web is based on a standard called X.509. When we mention SSL certificates, we usually mean X.509 certificates used in a TLS connection, such as that created by HTTPS.

X.509 is the ITU-T standard used to represent certificates, and has been chosen to be the standard used in the TLS protocol. The standard doesn't only define the binary structure of the certificate itself, but it also defines procedures to revoke the certificates, and establishes a hierarchical system of certification known as certificate path, or certificate chain.

The structure of an X.509 certificate is expressed using ASN.1, a notation used natively by the PEM format (discussed here). You can read the full specification in RFC 2459, in particular Section 4 "Certificate and Certificate Extensions Profile". I will refer to this later when I will have a look at a real certificate.

How are certificates related to HTTPS?¶

Before I discuss how certificates solve the problem of identity (or ownership of a public key), let's clarify the relationship between them and HTTPS.

HTTPS stands for HTTP Secure, and the core of the protocol consists of running HTTP over TLS. When we access a web site with HTTPS the browser first establishes a TLS connection with the server and then communicates with it using pure HTTP. This means that the whole HTTP protocol is encrypted, as the secure channel is established outside it, and also means that, aside from the different URI scheme https:// instead of http://, there are no differences between the two protocols.

Certificates come into play when the browser establishes the TLS connection, which is why you need to set-up HTTPS as part of your infrastructure and not in your web application. By the time the HTTP requests reach your application they are already decrypted and accessible in plain text, as the HTTP protocol mandates. We usually say that we "terminate TLS" when a component of our infrastructure manages certificates and decrypts HTTPS into HTTP.

How do certificates work?¶

The X.509 standard establishes entities called Certificate Authorities (CAs), and creates a hierarchy of trust called chain between them. The idea is that there is a set of entities that are trusted worldwide by operating systems, browsers, and other network-related software, and that these entities can trust other entities, thus creating a trust network.

While the market of Certificate Authorities is dominated by three major commercial players (see the usage statistics)there are approximately 100 organisations operating worldwide, among which some non-profit ones. Not all of these are trusted by all operating systems or browsers, though.

The set of CAs trusted by an organisation is called root program. The Mozilla community runs a program that is independent from the hardware/software platform, aptly called Mozilla's CA Certificate Program and uses data contained in the Common CA Database (CCADB). Private companies such as Microsoft, Apple, and Oracle run their own root programs and software running on the respective platforms (Windows, macOS/iOS, Java) can decide to trust the CAs provided by those programs.

In the open-source world, the Mozilla root program is by far the most influential and important source of information, being used by other software packages and Linux distributions.

It is possible to create certificates that are not signed by any CA, and these are called self-signed certificates. Such certificates can be used with any software that relies on certificates, but it requires such a software to disable certificate checking with the Certificate Authorities. Self-signed certificates are obviously useful for testing purposes, but there are scenarios in which it might be desirable not to rely on the CAs and establish a private network of trust.

Example: CA root certificate¶

The certificates for root CAs that are part of the Mozilla root program can be retrieved from the Common CA Database web page, or can be seen in the Firefox source code directly. On a running Firefox browser you can open the Privacy & Security menu and click on "View Certificates" at the bottom of the page. The CAs are listed under the tab "Authorities".

The interesting thing you can do here is to export a CA certificate. If you do it Firefox will save it in a file with extension .crt, that contains data in PEM format. I exported the certificate for Amazon Root CA 1 and I ended up with the file AmazonRootCA1.crt. If, instead of exporting, you view the certificate, you will end up in a page that allows you to download the certificate and the chain, both in PEM format, in files with the extension .pem. As you see, you are not the only one who is confused.

I described the PEM format in a post on RSA keys so I won't repeat here the whole discussion about it. The RFC 7468 ("Textual Encodings of PKIX, PKCS, and CMS Structures") describes certificates in section 5. Section 4 mentions the module id-pkix1-e for Certificate, CertificateList, and SubjectPublicKeyInfo RFC 5280 ("Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile").

The identifier id-pkix1-e is part of a registry of objects to be used in ASN.1 data created in the framework of the Public-Key Infrastructure using X.509 (PKIX) Working Group, that defined the infrastructure around the X.509 certificates system. Basically it's a standard way to identify binary objects and their structure. You can see a full list of all the objects in RFC 7299 ("Object Identifier Registry for the PKIX Working Group"). Not a very exciting one to read, if you ask me.

I can dump the content of the Amazon Root CA 1 certificate with OpenSSL

$ openssl asn1parse -inform pem -in amazon-root-ca-1.pem
    0:d=0  hl=4 l= 833 cons: SEQUENCE
    4:d=1  hl=4 l= 553 cons: SEQUENCE
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=  19 prim: INTEGER           :066C9FCF99BF8C0A39E2F0788A43E696365BCA
   34:d=2  hl=2 l=  13 cons: SEQUENCE
   36:d=3  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
   47:d=3  hl=2 l=   0 prim: NULL
   49:d=2  hl=2 l=  57 cons: SEQUENCE
   51:d=3  hl=2 l=  11 cons: SET
   53:d=4  hl=2 l=   9 cons: SEQUENCE
   55:d=5  hl=2 l=   3 prim: OBJECT            :countryName
   60:d=5  hl=2 l=   2 prim: PRINTABLESTRING   :US
   64:d=3  hl=2 l=  15 cons: SET
   66:d=4  hl=2 l=  13 cons: SEQUENCE
   68:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   73:d=5  hl=2 l=   6 prim: PRINTABLESTRING   :Amazon
   81:d=3  hl=2 l=  25 cons: SET
   83:d=4  hl=2 l=  23 cons: SEQUENCE
   85:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   90:d=5  hl=2 l=  16 prim: PRINTABLESTRING   :Amazon Root CA 1
  108:d=2  hl=2 l=  30 cons: SEQUENCE
  110:d=3  hl=2 l=  13 prim: UTCTIME           :150526000000Z
  125:d=3  hl=2 l=  13 prim: UTCTIME           :380117000000Z
  140:d=2  hl=2 l=  57 cons: SEQUENCE
  142:d=3  hl=2 l=  11 cons: SET
  144:d=4  hl=2 l=   9 cons: SEQUENCE
  146:d=5  hl=2 l=   3 prim: OBJECT            :countryName
  151:d=5  hl=2 l=   2 prim: PRINTABLESTRING   :US
  155:d=3  hl=2 l=  15 cons: SET
  157:d=4  hl=2 l=  13 cons: SEQUENCE
  159:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  164:d=5  hl=2 l=   6 prim: PRINTABLESTRING   :Amazon
  172:d=3  hl=2 l=  25 cons: SET
  174:d=4  hl=2 l=  23 cons: SEQUENCE
  176:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  181:d=5  hl=2 l=  16 prim: PRINTABLESTRING   :Amazon Root CA 1
  199:d=2  hl=4 l= 290 cons: SEQUENCE
  203:d=3  hl=2 l=  13 cons: SEQUENCE
  205:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption
  216:d=4  hl=2 l=   0 prim: NULL
  218:d=3  hl=4 l= 271 prim: BIT STRING
  493:d=2  hl=2 l=  66 cons: cont [ 3 ]
  495:d=3  hl=2 l=  64 cons: SEQUENCE
  497:d=4  hl=2 l=  15 cons: SEQUENCE
  499:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Basic Constraints
  504:d=5  hl=2 l=   1 prim: BOOLEAN           :255
  507:d=5  hl=2 l=   5 prim: OCTET STRING      [HEX DUMP]:30030101FF
  514:d=4  hl=2 l=  14 cons: SEQUENCE
  516:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Key Usage
  521:d=5  hl=2 l=   1 prim: BOOLEAN           :255
  524:d=5  hl=2 l=   4 prim: OCTET STRING      [HEX DUMP]:03020186
  530:d=4  hl=2 l=  29 cons: SEQUENCE
  532:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Subject Key Identifier
  537:d=5  hl=2 l=  22 prim: OCTET STRING      [HEX DUMP]:04148418CC8534ECBC0C94942E08599CC7B2104E0A08
  561:d=1  hl=2 l=  13 cons: SEQUENCE
  563:d=2  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
  574:d=2  hl=2 l=   0 prim: NULL
  576:d=1  hl=4 l= 257 prim: BIT STRING

Let's read part of it using the aforementioned section 4 of RFC 5280.

The signed certificate is a sequence of three main components

   Certificate  ::=  SEQUENCE  {
        tbsCertificate       TBSCertificate,
        signatureAlgorithm   AlgorithmIdentifier,
        signatureValue       BIT STRING  }

and the TBSCertificate structure represents the unsigned certificate (TBS = To Be Signed)

TBSCertificate  ::=  SEQUENCE  {
        version         [0]  EXPLICIT Version DEFAULT v1,
        serialNumber         CertificateSerialNumber,
        signature            AlgorithmIdentifier,
        issuer               Name,
        validity             Validity,
        subject              Name,
        subjectPublicKeyInfo SubjectPublicKeyInfo,
        issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
                             -- If present, version MUST be v2 or v3
        subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
                             -- If present, version MUST be v2 or v3
        extensions      [3]  EXPLICIT Extensions OPTIONAL
                             -- If present, version MUST be v3
        }

Comparing this with the output of OpenSSL we can find fields such as version

   10:d=3  hl=2 l=   1 prim: INTEGER           :02

which according to the documentation is 3 (binary 02). Many values are of type PRINTABLESTRING, so they are readable already in the ASN.1 dump.

The validity of the certificate is

  110:d=3  hl=2 l=  13 prim: UTCTIME           :150526000000Z
  125:d=3  hl=2 l=  13 prim: UTCTIME           :380117000000Z

and following section 4.1.2.5.1 of the RFC we find out that the certificate is valid between 26 May 2015 and 17 Jan 2038. You can easily read these values in the certificate page in the browser without getting an headache trying to decode ASN.1.

The CA signed the certificate using a certain algorithm. The algorithm identifier is repeated twice, first in the structure Certificate (signatureAlgorithm AlgorithmIdentifier) and then in the structure TBSCertificate (signature AlgorithmIdentifier). The two fields must have the same value.

   34:d=2  hl=2 l=  13 cons: SEQUENCE
   36:d=3  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
   47:d=3  hl=2 l=   0 prim: NULL

[...]

  561:d=1  hl=2 l=  13 cons: SEQUENCE
  563:d=2  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
  574:d=2  hl=2 l=   0 prim: NULL

For this certificate, the algorithm used by Amazon is sha256WithRSAEncryption. This label is described in RFC 4055 ("Additional Algorithms and Identifiers for RSA Cryptography for use in the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile") as "PKCS #1 version 1.5 signature algorithm with SHA-256". The specific algorithm can be found in RFC 2313 ("PKCS #1: RSA Encryption Version 1.5"). As the name of the algorithm suggests, the certificate is first digested with SHA-256 and then encrypted using RSA and the private key of the signer.

Speaking of keys, the public key the CA used for the certificate can be found in the field subjectpublickeyinfo, which is again made of a field type AlgorithmIdentifier and a bit string with the value of the key. In this case the fields are

  205:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption
  216:d=4  hl=2 l=   0 prim: NULL
  218:d=3  hl=4 l= 271 prim: BIT STRING

The algorithm rsaEncryption is described in RFC 3279 ("Algorithms and Identifiers for the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"), section 2.3.1 as

      RSAPublicKey ::= SEQUENCE {
         modulus            INTEGER,    -- n
         publicExponent     INTEGER  }  -- e

(sic) or in RFC 8017 ("PKCS #1: RSA Cryptography Specifications Version 2.2")

RSAPublicKey ::= SEQUENCE {
    modulus           INTEGER,  -- n
    publicExponent    INTEGER   -- e
}

We can then use the option -strparse of the module asn1parse to find the actual values

$ openssl asn1parse -inform pem -in amazon-root-ca-1.pem -strparse 218
    0:d=0  hl=4 l= 266 cons: SEQUENCE
    4:d=1  hl=4 l= 257 prim: INTEGER           :B2788071CA78D5E371AF478050747D6ED8D78876F4
9968F7582160F97484012FAC022D86D3A0437A4EB2A4D036BA01BE8DDB48C80717364CF4EE8823C73EEB37F5B5
19F84968B0DED7B976381D619EA4FE8236A5E54A56E445E1F9FDB416FA74DA9C9B35392FFAB02050066C7AD080
B2A6F9AFEC47198F503807DCA2873958F8BAD5A9F948673096EE94785E6F89A351C0308666A14566BA54EBA3C3
91F948DCFFD1E8302D7D2D747035D78824F79EC4596EBB738717F2324628B843FAB71DAACAB4F29F240E2D4BF7
715C5E69FFEA9502CB388AAE50386FDBFB2D621BC5C71E54E177E067C80F9C8723D63F40207F2080C4804C3E3B
24268E04AE6C9AC8AA0D
  265:d=1  hl=2 l=   3 prim: INTEGER           :010001

As we already saw for RSA keys), OpenSSL has a specific module for important structures, and the X.509 certificates are definitely worth a module aptly called x509. using that we can easily decode any certificate

$ openssl x509 -inform pem -in amazon-root-ca-1.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            06:6c:9f:cf:99:bf:8c:0a:39:e2:f0:78:8a:43:e6:96:36:5b:ca
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Amazon, CN = Amazon Root CA 1
        Validity
            Not Before: May 26 00:00:00 2015 GMT
            Not After : Jan 17 00:00:00 2038 GMT
        Subject: C = US, O = Amazon, CN = Amazon Root CA 1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b2:78:80:71:ca:78:d5:e3:71:af:47:80:50:74:
                    7d:6e:d8:d7:88:76:f4:99:68:f7:58:21:60:f9:74:
                    84:01:2f:ac:02:2d:86:d3:a0:43:7a:4e:b2:a4:d0:
                    36:ba:01:be:8d:db:48:c8:07:17:36:4c:f4:ee:88:
                    23:c7:3e:eb:37:f5:b5:19:f8:49:68:b0:de:d7:b9:
                    76:38:1d:61:9e:a4:fe:82:36:a5:e5:4a:56:e4:45:
                    e1:f9:fd:b4:16:fa:74:da:9c:9b:35:39:2f:fa:b0:
                    20:50:06:6c:7a:d0:80:b2:a6:f9:af:ec:47:19:8f:
                    50:38:07:dc:a2:87:39:58:f8:ba:d5:a9:f9:48:67:
                    30:96:ee:94:78:5e:6f:89:a3:51:c0:30:86:66:a1:
                    45:66:ba:54:eb:a3:c3:91:f9:48:dc:ff:d1:e8:30:
                    2d:7d:2d:74:70:35:d7:88:24:f7:9e:c4:59:6e:bb:
                    73:87:17:f2:32:46:28:b8:43:fa:b7:1d:aa:ca:b4:
                    f2:9f:24:0e:2d:4b:f7:71:5c:5e:69:ff:ea:95:02:
                    cb:38:8a:ae:50:38:6f:db:fb:2d:62:1b:c5:c7:1e:
                    54:e1:77:e0:67:c8:0f:9c:87:23:d6:3f:40:20:7f:
                    20:80:c4:80:4c:3e:3b:24:26:8e:04:ae:6c:9a:c8:
                    aa:0d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                84:18:CC:85:34:EC:BC:0C:94:94:2E:08:59:9C:C7:B2:10:4E:0A:08
    Signature Algorithm: sha256WithRSAEncryption
         98:f2:37:5a:41:90:a1:1a:c5:76:51:28:20:36:23:0e:ae:e6:
         28:bb:aa:f8:94:ae:48:a4:30:7f:1b:fc:24:8d:4b:b4:c8:a1:
         97:f6:b6:f1:7a:70:c8:53:93:cc:08:28:e3:98:25:cf:23:a4:
         f9:de:21:d3:7c:85:09:ad:4e:9a:75:3a:c2:0b:6a:89:78:76:
         44:47:18:65:6c:8d:41:8e:3b:7f:9a:cb:f4:b5:a7:50:d7:05:
         2c:37:e8:03:4b:ad:e9:61:a0:02:6e:f5:f2:f0:c5:b2:ed:5b:
         b7:dc:fa:94:5c:77:9e:13:a5:7f:52:ad:95:f2:f8:93:3b:de:
         8b:5c:5b:ca:5a:52:5b:60:af:14:f7:4b:ef:a3:fb:9f:40:95:
         6d:31:54:fc:42:d3:c7:46:1f:23:ad:d9:0f:48:70:9a:d9:75:
         78:71:d1:72:43:34:75:6e:57:59:c2:02:5c:26:60:29:cf:23:
         19:16:8e:88:43:a5:d4:e4:cb:08:fb:23:11:43:e8:43:29:72:
         62:a1:a9:5d:5e:08:d4:90:ae:b8:d8:ce:14:c2:d0:55:f2:86:
         f6:c4:93:43:77:66:61:c0:b9:e8:41:d7:97:78:60:03:6e:4a:
         72:ae:a5:d1:7d:ba:10:9e:86:6c:1b:8a:b9:59:33:f8:eb:c4:
         90:be:f1:b9

Now I'm pretty sure you want to kill me because I could have shown you this from the start. But I like to understand things, and the easy path doesn't always make everything clear. At any rate, here you have a way to read an X.509 certificate in PEM format.

Please note that in this certificate the Issuer and the Subject are the same entity, as this is a root certificate, which is signed by the same entity that creates it.

        Issuer: C = US, O = Amazon, CN = Amazon Root CA 1
[...]
        Subject: C = US, O = Amazon, CN = Amazon Root CA 1

Moreover, one of the version 3 extensions of the self-signed certificate is a basic constraint with the boolean CA set to true. It also has the extension Key Usage set to Digital Signature, Certificate Sign, CRL Sign, which means that the certificate can be used to sign other certificates.

Example: self-signed certificate¶

You can use OpenSSL to create a self-signed certificate using the module req that you would normally use to create certificate requests. As a self-signed certificate doesn't need approval, the module can directly output the certificate.

$ openssl req -x509 -newkey rsa:2048 -keyout self-signed-key.pem -out self-signed.pem -days 365 -nodes -subj '/CN=localhost'
Generating a RSA private key
....+++++
................+++++
writing new private key to 'self-signed-key.pem'
-----

(note that for simplicity's sake I specified the option -nodes that prevents the key to be protected with a password, but this is a bad practice). This command creates the two files I mentioned, self-signed-key.pem (the private key) and self-signed.pem.

We can read the certificate using the module x509

$ openssl x509 -inform pem -in self-signed.pem  -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            46:e5:2f:8e:42:82:43:b8:ac:88:cb:6d:0c:2f:71:28:a9:fe:00:ec
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = localhost
        Validity
            Not Before: Nov  3 00:23:34 2020 GMT
            Not After : Nov  3 00:23:34 2021 GMT
        Subject: CN = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b7:14:ef:3b:eb:8b:a9:40:18:c5:d2:eb:1d:4f:
                    5d:e4:a3:17:f3:df:ce:b7:d3:3f:52:58:eb:61:02:
                    a2:68:0a:cd:0f:97:ae:e0:a5:ac:a7:88:cf:a1:15:
                    0a:97:ca:e7:03:8a:a5:c0:66:38:ef:bb:59:4d:48:
                    17:db:a7:bd:fa:4b:50:2a:be:e9:5b:bb:59:65:71:
                    dc:99:73:9c:bc:4d:3b:42:97:91:e9:3b:1a:8a:9d:
                    cc:41:38:ba:8b:8f:df:65:ff:5b:1f:ef:8a:b7:c5:
                    93:07:ce:15:4c:13:72:78:59:64:9a:5b:95:20:b6:
                    b3:8e:aa:c3:29:c3:7f:28:39:43:81:59:e4:0f:26:
                    7c:3f:49:d2:06:05:d9:54:ab:09:65:96:01:cc:c2:
                    72:be:85:1f:40:ea:94:35:04:09:9d:87:eb:a1:90:
                    36:ce:d2:55:f9:ee:08:db:52:78:e8:70:d0:25:89:
                    13:8e:0f:9d:98:98:d1:4d:67:06:8f:8a:61:9e:3a:
                    73:89:aa:0a:0a:1b:05:a7:52:32:ef:1b:78:5a:5f:
                    4b:b6:c9:a7:4e:15:10:04:50:99:00:09:2f:60:8e:
                    aa:20:af:6b:ee:f5:60:0b:29:da:38:1c:b2:73:14:
                    99:a4:ee:5e:89:e6:77:0b:ba:cf:d3:5d:d7:a3:ea:
                    c4:bf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                64:7B:C1:FC:99:74:56:B7:82:D1:4F:E7:2D:94:77:1A:09:52:26:5C
            X509v3 Authority Key Identifier: 
                keyid:64:7B:C1:FC:99:74:56:B7:82:D1:4F:E7:2D:94:77:1A:09:52:26:5C

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         43:7b:0b:c8:98:b8:6f:72:af:39:4a:d9:76:ce:e3:9d:3a:c7:
         9f:14:b0:4f:20:0a:45:b3:b4:8c:e5:37:4c:bf:15:ad:8e:5c:
         45:4f:3e:b7:ef:8d:60:57:bb:6f:d9:5e:6a:d3:04:05:4a:ff:
         f2:66:b1:76:66:59:7e:24:89:0a:50:28:c9:d5:f5:7a:00:07:
         8a:79:9c:6e:53:43:66:e5:9a:10:d8:f8:e1:f2:c1:f1:17:d0:
         d2:9e:50:80:fe:2a:ca:08:b6:98:e9:b5:a4:82:23:31:45:35:
         33:da:2c:e3:fe:54:f2:bd:f2:61:91:f4:32:e3:7d:4c:3a:e5:
         3a:0f:cd:36:b0:8b:af:9f:8e:3d:0e:0b:a5:df:4a:3a:91:83:
         b3:b2:5f:3c:47:81:73:4f:a2:c1:49:06:75:17:25:fa:5a:8d:
         30:e5:55:7f:9c:3e:15:a8:b5:ab:f7:45:38:e3:76:8e:d4:0d:
         60:fc:42:17:3d:85:72:41:1d:53:9d:58:b0:e9:29:0c:e4:6b:
         14:c2:22:c4:d5:7b:de:36:da:df:d8:a0:4f:a4:0a:f2:3e:ca:
         7e:66:a6:10:38:97:24:73:5b:db:eb:0b:6c:a8:f8:37:15:2c:
         0e:b1:82:44:cc:fe:85:b0:cb:6c:26:4b:4a:70:33:dc:7e:f5:
         84:ba:07:db

As you can see this certificate has the same value in Issuer and Subject, as happened before for the Amazon Root one. It also has the flag CA set to true but it doesn't have the extension Key Usage meaning that this certificate can't be used to sign other certificates.

Example: this site's certificate¶

You can see TLS certificates and the chain of trust in action in this very website. Following the documentation of you browser (instructions for Firefox are here), you can see the certificate used by The Digital Cat. At the time of writing the blog is hosted on GitHub Pages, even tough I'm using a custom domain, and GitHub partnered with Let's Encrypt to provide certificates for such a configuration (details here).

Indeed, the certificate for thedigitalcatonline.com is provided by "Let's Encrypt Authority X3", which in turn is trusted by Digital Signature Trust Co. with its root CA "DST Root CA X3".

Let's have a look at the three certificates. The one for The Digital Cat is

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:93:02:bb:9a:c9:ed:a5:c3:d1:16:00:8b:15:76:af:e5:d9
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Oct 22 04:53:28 2020 GMT
            Not After : Jan 20 04:53:28 2021 GMT
        Subject: CN = www.thedigitalcatonline.com
[...]
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                63:4E:15:85:56:5A:A4:94:02:C2:16:42:A4:A5:97:9A:38:02:57:97
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:www.thedigitalcatonline.com
[...]

And you can see that this time the Subject is www.thedigitalcatonline.com, but the Issuer is Let's Encrypt Authority X3. The certificate provided by the organisation Let's Encrypt is

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Mar 17 16:40:46 2016 GMT
            Not After : Mar 17 16:40:46 2021 GMT
        Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
[...]
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            Authority Information Access: 
                OCSP - URI:http://isrg.trustid.ocsp.identrust.com
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier: 
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

            X509v3 Subject Key Identifier: 
                A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
[...]

Here, the Subject is Let's Encrypt Authority X3 (the Issuer of the previous certificate), and the Issuer is DST Root CA X3. Last, the certificate provided by the organisation Digital Signature Trust Co. is

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
[...]
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10
[...]

As happened for the certificate Amazon Root CA 1 that we discussed before, this one is self-signed, having the same value for Subject and Issuer.

How to verify certificates with OpenSSL¶

To verify if a certificate is valid we can use the module verify of OpenSSL. By default, OpenSSL doesn't trust anything, and verify relies on a default path in the system to find root certificates. You can see the path running

$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

On Ubuntu 20.04, the directory /usr/lib/ssl/certs is a symbolic link to /etc/ssl/certs that is installed by the package ca-certificates which is linked to the Mozilla's CA Certificate Program (details on that package can be found in the source code).

So, if a root certificate is included in the Mozilla program, it is trusted by OpenSSL

$ openssl verify amazon-root-ca-1.pem 
amazon-root-ca-1.pem: OK

while a self-signed certificate is not

$ openssl verify self-signed.pem 
CN = localhost
error 18 at 0 depth lookup: self signed certificate
error self-signed.pem: verification failed

A non-root certificate can be verified specifying which root certificate signed it. So, the certificate for this website is not trusted automatically

$ openssl verify www-thedigitalcatonline-com.pem 
CN = www.thedigitalcatonline.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error www-thedigitalcatonline-com.pem: verification failed

But it is verified specifying the certificate for Let's Encrypt that signed it

$ openssl verify -CAfile lets-encrypt-x3.pem www-thedigitalcatonline-com.pem 
www-thedigitalcatonline-com.pem: OK

because the certificate lets-encrypt-x3.pem is signed by DST_Root_CA_X3.pem which is included in the Mozilla program, and thus included in my Linux distribution.

If I remove the default certificates path OpenSSL doesn't accept the certificate for Let's Encrypt any more

$ openssl verify -no-CApath -CAfile lets-encrypt-x3.pem www-thedigitalcatonline-com.pem
C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
error 2 at 1 depth lookup: unable to get issuer certificate
error www-thedigitalcatonline-com.pem: verification failed

Low-level certificate validation process¶

Let's have a look at the signature process for x.509 certificates. The process depends on the specific algorithm used to sign the certificate, so I will use the certificate Amazon Root CA 1 as an example, leaving to the reader the investigation about other algorithms.

A signed certificate is made of two parts, the certificate itself and the signature. The signature contains an encrypted hash of the certificate. Being encrypted we can verify the signed using their public key, and once we decrypted it we can compare the hash with one that we create on the fly using the same algorithm.

For the Amazon root certificate, we know the signature algorithm and value from the output of openssl x509

$ openssl x509 -inform pem -in amazon-root-ca-1.pem  -noout -text
[...]
    Signature Algorithm: sha256WithRSAEncryption
         98:f2:37:5a:41:90:a1:1a:c5:76:51:28:20:36:23:0e:ae:e6:
         28:bb:aa:f8:94:ae:48:a4:30:7f:1b:fc:24:8d:4b:b4:c8:a1:
         97:f6:b6:f1:7a:70:c8:53:93:cc:08:28:e3:98:25:cf:23:a4:
         f9:de:21:d3:7c:85:09:ad:4e:9a:75:3a:c2:0b:6a:89:78:76:
         44:47:18:65:6c:8d:41:8e:3b:7f:9a:cb:f4:b5:a7:50:d7:05:
         2c:37:e8:03:4b:ad:e9:61:a0:02:6e:f5:f2:f0:c5:b2:ed:5b:
         b7:dc:fa:94:5c:77:9e:13:a5:7f:52:ad:95:f2:f8:93:3b:de:
         8b:5c:5b:ca:5a:52:5b:60:af:14:f7:4b:ef:a3:fb:9f:40:95:
         6d:31:54:fc:42:d3:c7:46:1f:23:ad:d9:0f:48:70:9a:d9:75:
         78:71:d1:72:43:34:75:6e:57:59:c2:02:5c:26:60:29:cf:23:
         19:16:8e:88:43:a5:d4:e4:cb:08:fb:23:11:43:e8:43:29:72:
         62:a1:a9:5d:5e:08:d4:90:ae:b8:d8:ce:14:c2:d0:55:f2:86:
         f6:c4:93:43:77:66:61:c0:b9:e8:41:d7:97:78:60:03:6e:4a:
         72:ae:a5:d1:7d:ba:10:9e:86:6c:1b:8a:b9:59:33:f8:eb:c4:
         90:be:f1:b9

You can see the signed certificate binary values with cat amazon-root-ca-1.pem | tail -n+2 | head -n-1 | base64 -di | hexdump -ve '/1 "%02x "' -e '2/8 "\n"'. While we can recognise the signature in the last 256 bytes we can't easily separate the bytes with the signature algorithm. If we open the signed certificate with an ASN.1 parser, instead, we can easily find the binary value of the certificate part

30 82 03 41 30 82 02 29 a0 03 02 01 02 02 13 06
6c 9f cf 99 bf 8c 0a 39 e2 f0 78 8a 43 e6 96 36
5b ca 30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05
00 30 39 31 0b 30 09 06 03 55 04 06 13 02 55 53
31 0f 30 0d 06 03 55 04 0a 13 06 41 6d 61 7a 6f
6e 31 19 30 17 06 03 55 04 03 13 10 41 6d 61 7a
6f 6e 20 52 6f 6f 74 20 43 41 20 31 30 1e 17 0d
31 35 30 35 32 36 30 30 30 30 30 30 5a 17 0d 33
38 30 31 31 37 30 30 30 30 30 30 5a 30 39 31 0b
30 09 06 03 55 04 06 13 02 55 53 31 0f 30 0d 06
03 55 04 0a 13 06 41 6d 61 7a 6f 6e 31 19 30 17
06 03 55 04 03 13 10 41 6d 61 7a 6f 6e 20 52 6f
6f 74 20 43 41 20 31 30 82 01 22 30 0d 06 09 2a
86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30
82 01 0a 02 82 01 01 00 b2 78 80 71 ca 78 d5 e3
71 af 47 80 50 74 7d 6e d8 d7 88 76 f4 99 68 f7
58 21 60 f9 74 84 01 2f ac 02 2d 86 d3 a0 43 7a
4e b2 a4 d0 36 ba 01 be 8d db 48 c8 07 17 36 4c
f4 ee 88 23 c7 3e eb 37 f5 b5 19 f8 49 68 b0 de
d7 b9 76 38 1d 61 9e a4 fe 82 36 a5 e5 4a 56 e4
45 e1 f9 fd b4 16 fa 74 da 9c 9b 35 39 2f fa b0
20 50 06 6c 7a d0 80 b2 a6 f9 af ec 47 19 8f 50
38 07 dc a2 87 39 58 f8 ba d5 a9 f9 48 67 30 96
ee 94 78 5e 6f 89 a3 51 c0 30 86 66 a1 45 66 ba
54 eb a3 c3 91 f9 48 dc ff d1 e8 30 2d 7d 2d 74
70 35 d7 88 24 f7 9e c4 59 6e bb 73 87 17 f2 32
46 28 b8 43 fa b7 1d aa ca b4 f2 9f 24 0e 2d 4b
f7 71 5c 5e 69 ff ea 95 02 cb 38 8a ae 50 38 6f
db fb 2d 62 1b c5 c7 1e 54 e1 77 e0 67 c8 0f 9c
87 23 d6 3f 40 20 7f 20 80 c4 80 4c 3e 3b 24 26
8e 04 ae 6c 9a c8 aa 0d 02 03 01 00 01 a3 42 30
40 30 0f 06 03 55 1d 13 01 01 ff 04 05 30 03 01
01 ff 30 0e 06 03 55 1d 0f 01 01 ff 04 04 03 02
01 86 30 1d 06 03 55 1d 0e 04 16 04 14 84 18 cc
85 34 ec bc 0c 94 94 2e 08 59 9c c7 b2 10 4e 0a
08

The signature algorithm part is

30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05 00
03 82 01 01 00

and the ASN.1 parser tells us that those bytes represent an OBJECT IDENTIFIER which value is 2.16.840.1.101.3.4.2.1. Now, object identifiers are not complicated per se, they are just a way to identify algorithms and other well known components in ASN.1 structures. The description of the field signatureAlgorithm of an x.509 certificate mentions three other RFCs that contains descriptions of the available algorithms. In particular, RFC 4055 contains the description of PKCS #1 one-way hash functions, one of which is

id-sha256  OBJECT IDENTIFIER  ::=  { joint-iso-itu-t(2)
                     country(16) us(840) organization(1) gov(101)
                     csor(3) nistalgorithm(4) hashalgs(2) 1 }

You can see the values in the object identifier between parentheses. Since these are PKCS #1 (a.k.a. RSA) has functions, OpenSSL identifies it as sha256WithRSAEncryption (see again RFC 4055).

RSA encryption is described in RFC 2313 ("PKCS #1: RSA Encryption Version 1.5") and the signature algorithm based on RSA is described there in section 10. In particular, section 10.2 details the verification process, which is the one we are interested in. The steps are

• Bit-string-to-octet-string conversion of the signature
• RSA decryption
• Digest decoding (ASN.1)
• Message digesting and comparison

As for the signature conversion, the sentence

Specifically, assuming that the length in bits of the
signature S is a multiple of eight, the first bit of the signature
shall become the most significant bit of the first octet of the
encrypted data, and so on through the last bit of the signature,
which shall become the least significant bit of the last octet of the
encrypted data.

is a very verbose way to say that the signature is big-endian.

So, the hexadecimal value of the signature is

98f2375a4190a11ac57651282036230eaee628bbaaf894ae48a4307f1bfc248d
4bb4c8a197f6b6f17a70c85393cc0828e39825cf23a4f9de21d37c8509ad4e9a
753ac20b6a897876444718656c8d418e3b7f9acbf4b5a750d7052c37e8034bad
e961a0026ef5f2f0c5b2ed5bb7dcfa945c779e13a57f52ad95f2f8933bde8b5c
5bca5a525b60af14f74befa3fb9f40956d3154fc42d3c7461f23add90f48709a
d9757871d1724334756e5759c2025c266029cf2319168e8843a5d4e4cb08fb23
1143e843297262a1a95d5e08d490aeb8d8ce14c2d055f286f6c49343776661c0
b9e841d7977860036e4a72aea5d17dba109e866c1b8ab95933f8ebc490bef1b9

And reading the field Subject Public Key Info of the certificate we find the public key. Remember that this is a root certificate, so it is signed using the same key that it contains, which is not true in general.

The public key's modulus is

b2788071ca78d5e371af478050747d6ed8d78876f49968f7582160f97484012f
ac022d86d3a0437a4eb2a4d036ba01be8ddb48c80717364cf4ee8823c73eeb37
f5b519f84968b0ded7b976381d619ea4fe8236a5e54a56e445e1f9fdb416fa74
da9c9b35392ffab02050066c7ad080b2a6f9afec47198f503807dca2873958f8
bad5a9f948673096ee94785e6f89a351c0308666a14566ba54eba3c391f948dc
ffd1e8302d7d2d747035d78824f79ec4596ebb738717f2324628b843fab71daa
cab4f29f240e2d4bf7715c5e69ffea9502cb388aae50386fdbfb2d621bc5c71e
54e177e067c80f9c8723d63f40207f2080c4804c3e3b24268e04ae6c9ac8aa0d

and the exponent is 0x10001 (default choice).

RSA public-key signature decryption is performed with signature ^ exponent mod modulus, and this operation returns

1fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
fffffffffffffffffffff003031300d0609608648016503040201050004206fc
4b8ac3d2b52c08baf56255e43d22c762962e4facab01ace16d48ec008be0a

Once the padding is removed, we are left with an ASN.1 binary structure that represents the digest

DigestInfo ::= SEQUENCE {
  digestAlgorithm DigestAlgorithmIdentifier,
  digest Digest }

(see RFC 2313 - Section 10.1.2)

The value of Digest can be extracted with an ASN.1 parser or by taking the last 256 bits and is 6fc4b8ac3d2b52c08baf56255e43d22c762962e4facab01ace16d48ec008be0a.

At this point we need to process the certificate bytes (without signature) with the SHA-256 hash function and we will find a matching value of 6fc4b8ac3d2b52c08baf56255e43d22c762962e4facab01ace16d48ec008be0a.

This process (for the specific case of this certificate) can be easily done in Python

from cryptography import x509
from hashlib import sha256

certificate_pem_file = "amazon-root-ca-1.pem"

with open(certificate_pem_file, "rb") as f:
    certificate_pem = f.read()

certificate = x509.load_pem_x509_certificate(certificate_pem)

modulus = certificate.public_key().public_numbers().n
exponent = certificate.public_key().public_numbers().e

signature = int.from_bytes(certificate.signature, "big")

verification = pow(signature, exponent, modulus)

digest = bytes().fromhex(str(hex(verification))[-64:])

calculated_digest = sha256(certificate.tbs_certificate_bytes)

print(digest.hex() == calculated_digest.hexdigest())

This is arguably not the best Python code ever, but it's a simple way to demonstrate the process. As I said, this is far from being general, as it assumes the signature is sha256WithRSAEncryption, which might not be the case.

What I showed you here is what happens when we validate a root certificate. When we validate a non-root certificate the process is exactly the same (taking into account that the algorithms involved might be different), only the public key used to sign the certificate doesn't come from the certificate itself, but from the signer one. So, in the case of this blog, the certificate for www.thedigitalcat.com has a signature encrypted with the public key of Let's Encrypt. And the certificate for Let's Encrypt will be signed using the public key of Digital Signature Trust Co. This is what creates the chain of trust.

Algorithms used by root certificates¶

A quick scan of the certificates that are part of the Mozilla program reveals that the vast majority of them is using RSA to self-sign them

$ for i in /etc/ssl/certs/*.pem; do openssl x509 -inform pem -in ${i} -noout -text | grep -E "Public Key Algorithm"; done | sort | uniq -c
     25             Public Key Algorithm: id-ecPublicKey
    114             Public Key Algorithm: rsaEncryption

while part of them are using id-ecPublicKey which is the identifier of elliptic curves algorithms.

When it comes to signature algorithms, instead, there is more variety

$ for i in /etc/ssl/certs/*.pem; do openssl x509 -inform pem -in ${i} -noout -text | grep -E "^    Signature Algorithm"; done | sort | uniq -c
      7     Signature Algorithm: ecdsa-with-SHA256
     18     Signature Algorithm: ecdsa-with-SHA384
     47     Signature Algorithm: sha1WithRSAEncryption
     57     Signature Algorithm: sha256WithRSAEncryption
      9     Signature Algorithm: sha384WithRSAEncryption
      1     Signature Algorithm: sha512WithRSAEncryption

Even here, elliptic curves are slowly being adopted.

AWS components related to certificates¶

If you are using AWS, you can create certificates with ACM, the AWS Certificate Manager. Such certificates cannot be downloaded, they can only be attached to other AWS components. For this reason, the generation process reuire you to create any request, as you might have to do with other authorities. Certificates created in the ACM are free.

Certificates created in the ACM can be attached to several AWS components, most notably Load Balancers, CloudFront, and API Gateway.

Traditionally, load balancers are the place where TLS is terminated for HTTPS, requiring a connection to port 443. While Application Load Balancers can do that, in 2019 AWS announced support for certificates in Network Load Balancers as well.

Let's encrypt¶

In an effort to push for HTTP encryption of any public server, the Internet Security Research Group founded in 2016 a non-profit CA named Let's Encrypt, which provides at no charge TLD certificates valid for 90 days. Such certificates can be renewed automatically as part of the setup (certbot) and represent a viable alternative to certificates issued by other CA, in particular for open source projects. This blog uses a certificate issued by Let's Encrypt (provided by GitHub Pages) and will thus expire in less than 3 months (but also automatically renewed).

Final words¶

I hope this post helped to clarify some of the most obscure points of certificates, that definitely bugged be when I first approached them. As always when standards are involved, the risk is to get lost in the myriad of documents where information is scattered, and not to realise that some (if not many) parts of the systems we run every day have a long history and thus a big burden of legacy code or nomenclature.

Resources¶

• The Wikipedia article on TLS
• The Wikipedia article on Certificate authority
• The Wikipedia article on X.509
• The Wikipedia article on Let's Encrypt
• OpenSSL documentation: asn1parse, x509, verify
• The Abstract Syntax Notation One ASN.1 interface description language
• RFC 2313 - "PKCS #1: RSA Encryption Version 1.5"
• RFC 2459 - "Internet X.509 Public Key Infrastructure Certificate and CRL Profile"
• RFC 3279 - "Algorithms and Identifiers for the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
• RFC 4055 - "Additional Algorithms and Identifiers for RSA Cryptography for use in the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
• RFC 5280 - "Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
• RFC 7299 - "Object Identifier Registry for the PKIX Working Group"
• RFC 7468 - "Textual Encodings of PKIX, PKCS, and CMS Structures"
• RFC 8017 - "PKCS #1: RSA Cryptography Specifications Version 2.2"
• RFC 8446 - "The Transport Layer Security (TLS) Protocol Version 1.3"
• pyca/cryptography - The Python pyca/cryptography package

November 04, 2020 10:00 PM UTC

Caktus Consulting Group

My Favorite Technical Blogs and Mailing Lists

November 04, 2020 07:16 PM UTC

Codementor

Introduction to Scrapy: Web Scraping in Python

Scrapy is an open-source web scraping framework, and it does a lot more than just a library. It manages requests, parses HTML webpages, collects data, and saves it to the desired format. Hence, you...

November 04, 2020 05:30 PM UTC

Real Python

Caching in Python Using the LRU Cache Strategy

 1import requests
 2
 3cache = dict()
 4
 5def get_article_from_server(url):
 6    print("Fetching article from server...")
 7    response = requests.get(url)
 8    return response.text
 9
10def get_article(url):
11    print("Getting article...")
12    if url not in cache:
13        cache[url] = get_article_from_server(url)
14
15    return cache[url]
16
17get_article("https://realpython.com/sorting-algorithms-python/")
18get_article("https://realpython.com/sorting-algorithms-python/")

$ pip install requests
$ python caching.py
Getting article...
Fetching article from server...
Getting article...

Read the full article at https://realpython.com/lru-cache-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

November 04, 2020 02:00 PM UTC

Stack Abuse

Gradient Descent in Python: Implementation and Theory

Introduction

This tutorial is an introduction to a simple optimization technique called gradient descent, which has seen major application in state-of-the-art machine learning models.

We'll develop a general purpose routine to implement gradient descent and apply it to solve different problems, including classification via supervised learning.

In this process, we'll gain an insight into the working of this algorithm and study the effect of various hyper-parameters on its performance. We'll also go over batch and stochastic gradient descent variants as examples.

What is Gradient Descent?

Gradient descent is an optimization technique that can find the minimum of an objective function. It is a greedy technique that finds the optimal solution by taking a step in the direction of the maximum rate of decrease of the function.

By contrast, Gradient Ascent is a close counterpart that finds the maximum of a function by following the direction of the maximum rate of increase of the function.

To understand how gradient descent works, consider a multi-variable function \(f(\textbf{w})\), where \(\textbf w = [w_1, w_2, \ldots, w_n]^T \). To find the \( \textbf{w} \) at which this function attains a minimum, gradient descent uses the following steps:

1. Choose an initial random value of \( \textbf{w} \)

2. Choose the number of maximum iterations T

3. Choose a value for the learning rate \( \eta \in [a,b] \)

4. Repeat following two steps until \(f\) does not change or iterations exceed T

   a.Compute: \( \Delta \textbf{w} = - \eta \nabla_\textbf{w} f(\textbf{w}) \)

   b. update \(\textbf{w} \) as: \(\textbf{w} \leftarrow \textbf{w} + \Delta \textbf{w} \)

   The left arrow symbol being a mathematically correct way to write an assignment statement.

Here \( \nabla_\textbf{w} f \) denotes the gradient of \(f\) as given by:
$$
\nabla_\textbf{w} f(\textbf{w}) =
\begin{bmatrix}
\frac{\partial f(\textbf{w})}{\partial w_1} \
\frac{\partial f(\textbf{w})}{\partial w_2} \
\vdots\
\frac{\partial f(\textbf{w})}{\partial w_n}
\end{bmatrix}
$$

Consider an example function of two variables \( f(w_1,w_2) = w_1^2+w_2^2 \), then at each iteration \( (w_1,w_2) \) is updated as:

$$
\begin {bmatrix}
w_1 \ w_2
\end {bmatrix} \leftarrow
\begin {bmatrix}
w_1 \ w_2
\end {bmatrix} - \eta
\begin {bmatrix}
2w_1 \ 2w_2
\end {bmatrix}
$$

The figure below shows how gradient descent works on this function.

The circles are the contours of this function. If we move along a contour, the function value would not change and would remain a constant.

This is opposed to the direction of the gradient, where the function changes at a maximum rate. Therefore the direction of the gradient of the function at any point is normal to the contour's tangent at that point.

In simple terms, the gradient can be taken as an arrow which points in the direction where the function changes the most.

Following the negative gradient direction would lead to points where the function value decreases at a maximum rate. The learning rate, also called the step size, dictates how fast or slow, we move along the direction of the gradient.

Adding Momentum

When using gradient descent, we run into the following problems:

1. Getting trapped in a local minimum, which is a direct consequence of this algorithm being greedy

2. Overshooting and missing the global optimum, this is a direct result of moving too fast along the gradient direction

3. Oscillation, this is a phenomenon that occurs when the function's value doesn't change significantly no matter the direction it advances. You can think of it as navigating a plateau, you're at the same height no matter where you go

To combat these problems, a momentum term \( \alpha \) is added to the expression for \(\Delta \textbf{w}\) to stabilize the rate of learning when moving towards the global optimum value.

Below, we use the superscript \(i\) to denote the iteration number:
$$
\Delta \textbf{w}^i = - \eta \nabla_\textbf{w} f(\textbf{w}^i) + \alpha \textbf{w}^{i-1}
$$

Implementing Gradient Descent in Python

Before we start writing the actual code for gradient descent, let's import some libraries we'll utilize to help us out:

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import sklearn.datasets as dt
from sklearn.model_selection import train_test_split

Now, with that out of the way, let's go ahead and define a gradient_descent() function. In this function, the loop ends when either:

1. The number of iterations exceeds a maximum value

2. The difference in function values between two successive iterations falls below a certain threshold

The parameters are updated at every iteration according to the gradient of the objective function.

The function will accept the following parameters:

• max_iterations: Maximum number of iterations to run

• threshold: Stop if the difference in function values between two successive iterations falls below this threshold

• w_init: Initial point from where to start gradient descent

• obj_func: Reference to the function that computes the objective function

• grad_func: Reference to the function that computes the gradient of the function

• extra_param: Extra parameters (if needed) for the obj_func and grad_func

• learning_rate: Step size for gradient descent. It should be in [0,1]

• momentum: Momentum to use. It should be in [0,1]

Also, the function will return:

• w_history: All points in space, visited by gradient descent at which the objective function was evaluated

• f_history: Corresponding value of the objective function computed at each point

# Make threshold a -ve value if you want to run exactly
# max_iterations.
def gradient_descent(max_iterations,threshold,w_init,
                     obj_func,grad_func,extra_param = [],
                     learning_rate=0.05,momentum=0.8):
    
    w = w_init
    w_history = w
    f_history = obj_func(w,extra_param)
    delta_w = np.zeros(w.shape)
    i = 0
    diff = 1.0e10
    
    while  i<max_iterations and diff>threshold:
        delta_w = -learning_rate*grad_func(w,extra_param) + momentum*delta_w
        w = w+delta_w
        
        # store the history of w and f
        w_history = np.vstack((w_history,w))
        f_history = np.vstack((f_history,obj_func(w,extra_param)))
        
        # update iteration number and diff between successive values
        # of objective function
        i+=1
        diff = np.absolute(f_history[-1]-f_history[-2])
    
    return w_history,f_history

Optimizing Functions with Gradient Descent

Now that we have a general purpose implementation of gradient descent, let's run it on our example 2D function \( f(w_1,w_2) = w_1^2+w_2^2 \) with circular contours.

The function has a minimum value of zero at the origin. Let's visualize the function first and then find its minimum value.

Visualizing the Objective Function f(x)

The visualize_fw() function below, generates 2500 equally spaced points on a grid and computes the function value at each point.

The function_plot() function displays all points in different colors, depending upon the value of \(f(\textbf w)\) at that point. All points at which the function's value is the same, have the same color:

def visualize_fw():
    xcoord = np.linspace(-10.0,10.0,50)
    ycoord = np.linspace(-10.0,10.0,50)
    w1,w2 = np.meshgrid(xcoord,ycoord)
    pts = np.vstack((w1.flatten(),w2.flatten()))
    
    # All 2D points on the grid
    pts = pts.transpose()
    
    # Function value at each point
    f_vals = np.sum(pts*pts,axis=1)
    function_plot(pts,f_vals)
    plt.title('Objective Function Shown in Color')
    plt.show()
    return pts,f_vals

# Helper function to annotate a single point
def annotate_pt(text,xy,xytext,color):
    plt.plot(xy[0],xy[1],marker='P',markersize=10,c=color)
    plt.annotate(text,xy=xy,xytext=xytext,
                 # color=color,
                 arrowprops=dict(arrowstyle="->",
                 color = color,
                 connectionstyle='arc3'))

# Plot the function
# Pts are 2D points and f_val is the corresponding function value
def function_plot(pts,f_val):
    f_plot = plt.scatter(pts[:,0],pts[:,1],
                         c=f_val,vmin=min(f_val),vmax=max(f_val),
                         cmap='RdBu_r')
    plt.colorbar(f_plot)
    # Show the optimal point
    annotate_pt('global minimum',(0,0),(-5,-7),'yellow')    

pts,f_vals = visualize_fw()

Running Gradient Descent with Different Hyper-parameters

Now it's time to run gradient descent to minimize our objective function. To call gradient_descent(), we define two functions:

• f(): Computes the objective function at any point w
• grad(): Computes the gradient at any point w

To understand the effect of various hyper-parameters on gradient descent, the function solve_fw() calls gradient_descent() with 5 iterations for different values of learning rate and momentum.

The function visualize_learning(), plots the values of \((w_1,w_2) \), with function values shown in different colors. The arrows in the plot make it easier to track which point was updated from the last:

# Objective function
def f(w,extra=[]):
    return np.sum(w*w)

# Function to compute the gradient
def grad(w,extra=[]):
    return 2*w

# Function to plot the objective function
# and learning history annotated by arrows
# to show how learning proceeded
def visualize_learning(w_history):  
    
    # Make the function plot
    function_plot(pts,f_vals)
    
    # Plot the history
    plt.plot(w_history[:,0],w_history[:,1],marker='o',c='magenta') 
    
    # Annotate the point found at last iteration
    annotate_pt('minimum found',
                (w_history[-1,0],w_history[-1,1]),
                (-1,7),'green')
    iter = w_history.shape[0]
    for w,i in zip(w_history,range(iter-1)):
        # Annotate with arrows to show history
        plt.annotate("",
                    xy=w, xycoords='data',
                    xytext=w_history[i+1,:], textcoords='data',
                    arrowprops=dict(arrowstyle='<-',
                            connectionstyle='angle3'))     
    
def solve_fw():
    # Setting up
    rand = np.random.RandomState(19)
    w_init = rand.uniform(-10,10,2)
    fig, ax = plt.subplots(nrows=4, ncols=4, figsize=(18, 12))
    learning_rates = [0.05,0.2,0.5,0.8]
    momentum = [0,0.5,0.9]
    ind = 1
    
    # Iteration through all possible parameter combinations
    for alpha in momentum:
        for eta,col in zip(learning_rates,[0,1,2,3]):
            plt.subplot(3,4,ind)        
            w_history,f_history = gradient_descent(5,-1,w_init, f,grad,[],eta,alpha)
            
            visualize_learning(w_history)
            ind = ind+1
            plt.text(-9, 12,'Learning Rate = '+str(eta),fontsize=13)
            if col==1:
                plt.text(10,15,'momentum = ' + str(alpha),fontsize=20)

    fig.subplots_adjust(hspace=0.5, wspace=.3)
    plt.show()

Let's run solve_fw() and see how the learning rate and momentum effect gradient descent:

solve_fw()

This example clarifies the role of both momentum and learning rate.

In the first plot, with zero momentum and learning rate set at 0.05, learning is slow and the algorithm does not reach the global minimum. Increasing the momentum speeds up learning as we can see from the plots in the first column. The other extreme is the last column, where the learning rate is kept high. This causes oscillations, which can be controlled to a certain extent by adding momentum.

The general guideline for gradient descent is to use small values of learning rate and higher values of momentum.

Gradient Descent for Minimizing Mean Square Error

Gradient descent is a nice and simple technique for minimizing the mean square error in a supervised classification or regression problem.

Suppose we are given \(m\) training examples \([x_{ij}]\) with \(i=1\ldots m \), where each example has \(n\) features, i.e., \(j=1\ldots n \). If the corresponding target and output values for each example are \(t_i\) and \(o_i\) respectively, then the mean square error function \(E\) (in this case our object function) is defined as:

$$
E = \frac{1}{m} \Sigma_{i=1}^m (t_i - o_i)^2
$$

Where the output \(o_i\) is determined by a weighted linear combination of inputs, given by:

$$
o_i = w_0 + w_1 x_{i1} + w_2 x_{i2} + \ldots + w_n x_{in}
$$

The unknown parameter in the above equation is the weight vector \(\textbf w = [w_0,w_1,\ldots,w_n]^T\).

The objective function in this case is the mean square error with a gradient given by:

$$
\nabla_{\textbf w}E(\textbf w) = -\Sigma_{i=1}^{m} (t_i - o_i) \textbf{x}_i
$$

Where \(x_{i}\) is the i-th example. or an array of features of size n.

All we need now is a function to compute the gradient and a function to compute the mean square error.

The gradient_descent() function can then be used as-is. Note that all training examples are processed together when computing the gradient. Hence, this version of gradient descent for updating weights is referred to as batch updating or batch learning:

# Input argument is weight and a tuple (train_data, target)
def grad_mse(w,xy):
    (x,y) = xy
    (rows,cols) = x.shape
    
    # Compute the output
    o = np.sum(x*w,axis=1)
    diff = y-o
    diff = diff.reshape((rows,1))    
    diff = np.tile(diff, (1, cols))
    grad = diff*x
    grad = -np.sum(grad,axis=0)
    return grad

# Input argument is weight and a tuple (train_data, target)
def mse(w,xy):
    (x,y) = xy
    
    # Compute output
    # keep in mind that wer're using mse and not mse/m
    # because it would be relevant to the end result
    o = np.sum(x*w,axis=1)
    mse = np.sum((y-o)*(y-o))
    mse = mse/2
    return mse    

Running Gradient Descent on OCR

To illustrate gradient descent on a classification problem, we have chosen the digits datasets included in sklearn.datasets.

To keep things simple, let's do a test run of gradient descent on a two-class problem (digit 0 vs. digit 1). The code below loads the digits and displays the first 10 digits. This gives us an idea of the nature of training points:

# Load the digits dataset with two classes
digits,target = dt.load_digits(n_class=2,return_X_y=True)
fig,ax = plt.subplots(nrows=1, ncols=10,figsize=(12,4),subplot_kw=dict(xticks=[], yticks=[]))

# Plot some images of digits
for i in np.arange(10):
    ax[i].imshow(digits[i,:].reshape(8,8),cmap=plt.cm.gray)   
plt.show()

We also need the method train_test_split from sklearn.model_selection to split the training data into a train and a test set. The code below runs gradient descent on the training set, learns the weights, and plots the mean square error at different iterations.

When running gradient descent, we'll keep learning rate and momentum very small as the inputs are not normalized or standardized. Also, the batch version of gradient descent requires a smaller learning rate:

# Split into train and test set
x_train, x_test, y_train, y_test = train_test_split(
                        digits, target, test_size=0.2, random_state=10)

# Add a column of ones to account for bias in train and test
x_train = np.hstack((np.ones((y_train.size,1)),x_train))
x_test  = np.hstack((np.ones((y_test.size,1)),x_test))

# Initialize the weights and call gradient descent
rand = np.random.RandomState(19)
w_init = rand.uniform(-1,1,x_train.shape[1])*.000001
w_history,mse_history = gradient_descent(100,0.1,w_init,
                              mse,grad_mse,(x_train,y_train),
                             learning_rate=1e-6,momentum=0.7)

# Plot the MSE
plt.plot(np.arange(mse_history.size),mse_history)
plt.xlabel('Iteration No.')
plt.ylabel('Mean Square Error')
plt.title('Gradient Descent on Digits Data (Batch Version)')
plt.show()

This looks great! Let's check the error rate of our OCR on the training and test data. Below is a small function to compute the error rate of classification, which is called on the training and test set:

# Returns error rate of classifier
# total miclassifications/total*100
def error(w,xy):
    (x,y) = xy
    o = np.sum(x*w,axis=1)
    
    #map the output values to 0/1 class labels
    ind_1 = np.where(o>0.5)
    ind_0 = np.where(o<=0.5)
    o[ind_1] = 1
    o[ind_0] = 0
    return np.sum((o-y)*(o-y))/y.size*100
    
train_error = error(w_history[-1],(x_train,y_train))
test_error = error(w_history[-1],(x_test,y_test))

print("Train Error Rate: " + "{:.2f}".format(train_error))
print("Test Error Rate: " + "{:.2f}".format(test_error))

Train Error Rate: 0.69
Test Error Rate: 1.39

Stochastic Gradient Descent in Python

In the previous section, we used the batch updating scheme for gradient descent.

Another version of gradient descent is the online or stochastic updating scheme, where each training example is taken one at a time for updating the weights.

Once all training examples are cycled through, we say that an epoch is completed. The training examples are shuffled before each epoch, for better results.

The code snippet below is a slight modification of the gradient_descent() function to incorporate its stochastic counterpart. This function takes the (training set, target) as a parameter instead of the extra parameter. The term 'iterations' has been renamed to 'epochs':

# (xy) is the (training_set,target) pair
def stochastic_gradient_descent(max_epochs,threshold,w_init,
                                obj_func,grad_func,xy,
                                learning_rate=0.05,momentum=0.8):
    (x_train,y_train) = xy
    w = w_init
    w_history = w
    f_history = obj_func(w,xy)
    delta_w = np.zeros(w.shape)
    i = 0
    diff = 1.0e10
    rows = x_train.shape[0]
    
    # Run epochs
    while  i<max_epochs and diff>threshold:
        # Shuffle rows using a fixed seed to reproduce the results
        np.random.seed(i)
        p = np.random.permutation(rows)
        
        # Run for each instance/example in training set
        for x,y in zip(x_train[p,:],y_train[p]):
            delta_w = -learning_rate*grad_func(w,(np.array([x]),y)) + momentum*delta_w
            w = w+delta_w
            
        i+=1
        w_history = np.vstack((w_history,w))
        f_history = np.vstack((f_history,obj_func(w,xy)))
        diff = np.absolute(f_history[-1]-f_history[-2])
        
    return w_history,f_history

Let's run the code to see how the results are for stochastic version of gradient descent:

rand = np.random.RandomState(19)
w_init = rand.uniform(-1,1,x_train.shape[1])*.000001
w_history_stoch,mse_history_stoch = stochastic_gradient_descent(
                                100,0.1,w_init,
                              mse,grad_mse,(x_train,y_train),
                             learning_rate=1e-6,momentum=0.7)

# Plot the MSE
plt.plot(np.arange(mse_history_stoch.size),mse_history_stoch)
plt.xlabel('Iteration No.')
plt.ylabel('Mean Square Error')
plt.title('Gradient Descent on Digits Data (Stochastic Version)')

plt.show()

Let's also check the error rate:

train_error_stochastic = error(w_history_stoch[-1],(x_train,y_train))
test_error_stochastic = error(w_history_stoch[-1],(x_test,y_test))

print("Train Error rate with Stochastic Gradient Descent: " + 
      "{:.2f}".format(train_error_stochastic))
print("Test Error rate with Stochastic Gradient Descent: "  
      + "{:.2f}".format(test_error_stochastic))

Train Error rate with Stochastic Gradient Descent: 0.35
Test Error rate with Stochastic Gradient Descent: 1.39

Comparing Batch and Stochastic Versions

Let's now compare both the batch and stochastic versions of gradient descent.

We'll fix the learning rate for both versions to the same value and vary momentum to see how fast they both converge. The initial weights and the stopping criteria for both algorithms remain the same:

fig, ax = plt.subplots(nrows=3, ncols=1, figsize=(10,3))

rand = np.random.RandomState(11)
w_init = rand.uniform(-1,1,x_train.shape[1])*.000001
eta = 1e-6
for alpha,ind in zip([0,0.5,0.9],[1,2,3]):
	w_history,mse_history = gradient_descent(
                                100,0.01,w_init,
                              mse,grad_mse,(x_train,y_train),
                             learning_rate=eta,momentum=alpha)

    w_history_stoch,mse_history_stoch = stochastic_gradient_descent(
                                100,0.01,w_init,
                              mse,grad_mse,(x_train,y_train),
                             learning_rate=eta,momentum=alpha)
    
    # Plot the MSE
    plt.subplot(130+ind)
    plt.plot(np.arange(mse_history.size),mse_history,color='green')
    plt.plot(np.arange(mse_history_stoch.size),mse_history_stoch,color='blue')
    plt.legend(['batch','stochastic'])
    
    # Display total iterations
    plt.text(3,-30,'Batch: Iterations='+
             str(mse_history.size) )
    plt.text(3,-45,'Stochastic: Iterations='+
             str(mse_history_stoch.size))
    plt.title('Momentum = ' + str(alpha))   
    
    # Display the error rates
    train_error = error(w_history[-1],(x_train,y_train))
    test_error = error(w_history[-1],(x_test,y_test))
    
    train_error_stochastic = error(w_history_stoch[-1],(x_train,y_train))
    test_error_stochastic = error(w_history_stoch[-1],(x_test,y_test))
    
    print ('Momentum = '+str(alpha))
    
    print ('\tBatch:')
    print ('\t\tTrain error: ' + "{:.2f}".format(train_error) )
    print ('\t\tTest error: ' + "{:.2f}".format(test_error) )
    
    print ('\tStochastic:')
    print ('\t\tTrain error: ' + "{:.2f}".format(train_error_stochastic) )
    print ('\t\tTest error: ' + "{:.2f}".format(test_error_stochastic) )
    
        
plt.show()

Momentum = 0
	Batch:
		Train error: 0.35
		Test error: 1.39
	Stochastic:
		Train error: 0.35
		Test error: 1.39
Momentum = 0.5
	Batch:
		Train error: 0.00
		Test error: 1.39
	Stochastic:
		Train error: 0.35
		Test error: 1.39
Momentum = 0.9
	Batch:
		Train error: 0.00
		Test error: 1.39
	Stochastic:
		Train error: 0.00
		Test error: 1.39

While there isn't a significant difference in the accuracy between the two versions of the classifier, the stochastic version is a clear winner when it comes to the speed of convergence. It takes fewer iterations to achieve the same result as its batch counterpart.

Conclusions

Gradient descent is a simple and easy to implement technique.

In this tutorial, we illustrated gradient descent on a function of two variables with circular contours. We then extended our example to minimize the mean square error in a classification problem and built a simple OCR system. We also discussed the stochastic version of gradient descent.

A general purpose function for implementing gradient descent was developed in this tutorial. We encourage the readers to use this function on different regression and classification problems, with different hyper-parameters, for a better understanding of its working.

November 04, 2020 01:30 PM UTC

Python Insider

Python 3.10.0a2 is now available for testing

The engines of the secret release manager machine have finished producing a new pre-release. Go get it here:

https://www.python.org/downloads/release/python-3100a2/ 

This is an early developer preview of Python 3.10

Major new features of the 3.10 series, compared to 3.9

Python 3.10 is still in development. This releasee, 3.10.0a2 is the second of six planned alpha releases.
Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process.
During the alpha phase, features may be added up until the start of the beta phase (2021-05-03) and, if necessary, may be modified or deleted up until the release candidate phase (2021-10-04). Please keep in mind that this is a preview release and its use is not recommended for production environments.

Many new features for Python 3.10 are still being planned and written. Among the new major
new features and changes so far:

• PEP 623 – Remove wstr from Unicode
• PEP 604 – Allow writing union types as X | Y
• PEP 612 – Parameter Specification Variables
• PEP 626 – Precise line numbers for debugging and other tools.
• bpo-38605: from __future__ import annotations (PEP 563) is now the default.
• (Hey, fellow core developer, if a feature you find important is missing from this list, let Pablo know.)

The next pre-release of Python 3.10 will be 3.10.0a3, currently scheduled for 2020-12-07.

More resources

• Online Documentation
• PEP 619, 3.10 Release Schedule
• Report bugs at https://bugs.python.org.
• Help fund Python and its community.

And now for something completely different

The cardinality (the number of elements) of infinite sets can be one of the most surprising results of set theory. For example, there are the same amount of even natural numbers than natural numbers (which can be even or odd). There is also the same amount of rational numbers than natural numbers. But on the other hand, there are more real numbers between 0 and 1 than natural numbers! All these sets have infinite cardinality but turn out that some of these infinities are bigger than others. These infinite cardinalities normally are represented using aleph numbers. Infinite sets are strange beasts indeed.

Your friendly release team,
Ned Deily @nad
Steve Dower @steve.dower
Pablo Galindo Salgado @pablogsal

November 04, 2020 09:59 AM UTC

NumFOCUS

JupyterCon 2020: Code of Conduct Reports

Following the reports to the NumFOCUS Code-of-Conduct committee on Jeremy Howard’s keynote at JupyterCon 2020, and the controversy that followed, the NumFOCUS Code of Conduct Committee issued a public apology to Jeremy Howard and escalated the case to the board of directors. The context In his keynote at JupyterCon 2020, Jeremy Howard gave a point-by-point rebuttal of […]

The post JupyterCon 2020: Code of Conduct Reports appeared first on NumFOCUS.

November 04, 2020 12:10 AM UTC

Matt Layman

User Authentication

In the previous Understand Django article, we learned about the structure of a Django application and how apps are the core components of a Django project. In this article, we’re going to dig into Django’s built-in user authentication system. We’ll see how Django makes your life easier by giving you tools to help your web application interact with the users of your site. From Browser To DjangoURLs Lead The WayViews On ViewsTemplates For User InterfacesUser Interaction With FormsStore Data With ModelsAdminister All The ThingsAnatomy Of An ApplicationUser Authentication Authentication And Authorization We need to start with some terms before we begin our study.

November 04, 2020 12:00 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
  • Planet Python Brasil
  • Planet Python Indonesia
  • Planet Python Poland
• Python Libraries
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Coder's Cat"
  • "CodersLegacy"
  • "Control F'd"
  • "Fredrik Håård's Blaag"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • 4teamwork
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AWeber Engineering
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Ahmed Bouchefra
  • Al-Ahmadgaid Asaad
  • Albert Hopkins
  • Alec Munro
  • Alex Clark
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amit Saha
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Ankit Chandawala
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • AppNeta Blog
  • Armin Ronacher
  • Arnav Khare
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Atul Varma -- Toolness
  • Audrey Roy Greenfeld
  • Automating OSINT
  • Awesome Python Applications
  • Baiju Muthukadan
  • Bajusz Tamás
  • Balthazar Rouberol
  • BangPypers
  • BeDjango
  • Ben Bass
  • Ben Rousch
  • Ben Tappin
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhavin Gandhi
  • Bhishan Bhandari
  • Bill Ward / AdminTome
  • BioPython News
  • Bit of Cheese
  • Blue Yonder Tech
  • BlueBream
  • Bojan Mihelac
  • Brad Lucas
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Harring
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catalin George Festila
  • Catherine Devlin
  • Checking and Sharing
  • Ching-Hwa Yu
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Barra
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • Christopher Lenz
  • Chuck Thier
  • ClusterHQ
  • Cobe.io
  • CodeGrades
  • CodeSnipers
  • Codementor
  • Coding Diet
  • Collin Winter
  • Continuum Analytics Blog
  • Continuum Analytics News
  • Continuum Blog
  • Core Software
  • Corey Gallon
  • Corey Goldberg
  • Corey Oordt
  • Cormoran Project
  • Craig Kerstiens
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Damián Avila
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Dariusz Suchojad
  • Data Community DC
  • Data School
  • DataCamp
  • Dataquest
  • Dave Beazley
  • David Caron
  • David Goodger
  • David Grant
  • David J C Beach
  • David MacIver
  • David Malcolm
  • David Marte
  • David Stanek
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diego Garcia
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • DreamHost
  • Duncan McGreggor
  • Dusty Phillips
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eko S. Wibowo
  • Eli Bendersky
  • Eniram Ltd.
  • Enthought
  • Eray Özkural (examachine)
  • Eric Holscher
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Even Wiik Thomassen
  • Evennia
  • Everyday Superpowers
  • Experienced Django
  • Exxact Corp
  • Fabio Zadrozny
  • Fazal Majid
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • Fredrik Lundh
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Gabriel Pettier
  • Galvanize
  • Gary Bernhardt
  • Gary Wilson
  • Geek Scrap
  • Georg Brandl
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • GoDjango
  • Gocept Weblog
  • Godson Gera
  • Gonçalo Margalho
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Gridarrow
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Guilherme Toti
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hany Fahim
  • Hector Garcia
  • Heikki Toivonen
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Humberto Rocha
  • Hynek Schlawack
  • IO Digital Sec
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Intellimath blog
  • Ionel Cristian Maries
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • IslandT
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Krstic
  • J. Pablo Fernández
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • James Mills
  • Jan-Carel Brand
  • Janusworx
  • Jarrod Millman
  • Jason Meyers
  • Jason Webb
  • Jaysinh Shukla
  • Jean-Louis Fuchs
  • Jean-Paul Calderone
  • Jeethu Rao
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Knupp
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Jacobsen
  • John Paulett
  • Jon Parise
  • Jonathan Dobson
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan Street
  • Jorge Niedbalski
  • Jorge Puente Sarrín
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Juan Rodríguez Monti
  • Juho Vepsäläinen
  • Julien Danjou
  • Julien Palard
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kate Editor
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Kodnito
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kracekumar Ramaraju
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kumar Vipin Yadav
  • Kushal Das
  • Károly Nagy
  • LAAC Technology
  • Laurence Rowe
  • Laurent Luce
  • Laurent Szyster
  • Learn PyQt
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lightning Fast Shop
  • Lintel Technologies
  • Lion Kimbro
  • Lionel Tan
  • ListenData
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Lucas Cimon
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Macken
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marc Kerins
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Mark Mruss
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mathieu Virbel
  • Mats Kindahl
  • Matt Harrison
  • Matt Layman
  • Matt Reid
  • Matt Wilkes
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Mattias Brändström
  • Mauveweb
  • Max Ischenko
  • Michael Bayer
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael Nelson
  • Michael Sparks
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikeal Rogers
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Morten W Petersen
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Nadia Alramli
  • Naomi Ceder
  • Natan Zabkar
  • Nathan Lemoine
  • Nathan Piccini Data Science Dojo Blog
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Amorim
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • NumFOCUS
  • Obey the Testing Goat
  • Ofosos
  • Oliver Andrich
  • Omaha Python Users Group
  • Ondřej Čertík
  • OpenHatch Python posts
  • PSF GSoC students blogs
  • Paolo Amoroso
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patricio Paez
  • Patrick Kennedy
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • PayPal Engineering Blog
  • Pedro Lima
  • Petecode
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philip Semanchuk
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Pierre Carbonnelle
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programando Ciência
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyATL Bytecode
  • PyBites
  • PyCarolinas
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyCon Australia
  • PyCon Pune
  • PyCon.DE & PyData Karlsruhe
  • PyPy Development
  • PyTennessee
  • PyTexas
  • Pypix
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Circle
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python Morsels
  • Python Open Mike
  • Python Piedmont Triad User Group
  • Python Software Foundation
  • Python Sweetness
  • Python Testing Cookbook
  • Python User Groups
  • Python for Beginners
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • PythonThreads
  • PythonTrainingHQ
  • Pythonicity
  • Pythonistas.dev
  • Pythonology
  • Python⇒Speed
  • Péter Szabó
  • Péter Zsoldos
  • Quansight Labs Blog
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Reinout van Rees
  • Rene Dudfield
  • Reuven Lerner
  • Richard Gomes
  • Richard Jones
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Zaremba
  • Robin Parmar
  • Robin Wilson
  • Roche Compaan
  • Rodrigo Araúj
  • Roman Imankulov
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • STX Next
  • Salim Fadhley
  • Salman Haq
  • Sandipan Dey
  • Sandro Tosi
  • Sayan Chowdhury
  • Scripting the web with Python
  • Sean McGrath
  • Sebastian Pölsterl
  • Sebastian Witowski
  • Selena Deckelmann
  • Semaphore Community
  • Senthil Kumaran
  • Server Density
  • Shannon -jj Behrens
  • ShiningPanda
  • Shopkick Tech Blog
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stacks
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Petrea
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Dower
  • Steve Holden
  • Steven Klass
  • Stijn Debrouwere
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Sumith - Blog about SymPy/Python
  • Swaroop C H
  • Swisscom ICT
  • Sylvain Hellegouarch
  • Talk Python to Me
  • Tarek Ziade
  • Taylor Edmiston
  • TechBeamers Python
  • Techiediaries - Django
  • Ted Leung
  • Tennessee Leeuwenburg
  • Terri Oda
  • Terry Jones
  • Terry Peppers
  • Test and Code
  • TestDriven.io
  • The Artificial Intelligence Cookbook
  • The Code Bits
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • The R Trader
  • The Three of Wands
  • TheAutomatic.net
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Thomi Richards
  • Tiago Montes
  • Tibo Beijen
  • Tim Arnold / reachtim
  • Tim Gilbert
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim van der Linden
  • Tobias Ivarsson
  • Toes to the Edge
  • Tom Christie
  • Tomasz Ducin
  • Tomasz Früboes
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Two Bit Arcade
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinayak Mehta
  • Vinta Software
  • Virgil Dupras
  • Vladimir Iakolev
  • Vladimir Perić
  • Wallaroo Labs
  • Wallix
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Wichert Akkerman
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • William Thompson
  • Wing Tips
  • Wingware
  • Wraithan
  • Wyatt Baldwin
  • Yaco
  • Yaniv Aknin
  • Yann Larrivée
  • Yasoob Khalid
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero to Mastery
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • codingdirectional
  • eGenix.com
  • eRambler
  • gamingdirectional
  • hypothesis.works articles
  • kdev-python
  • leftmouseclickin
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • spikelantern
  • testmon
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • To request addition or removal:
    Open an issue on github
    or e-mail planet at python.org
    (note, responses can take up to a few days)