Articles

Psycopg 3.3 released

2025-12-01T00:00:00Z

We have released Psycopg 3.3 — and you should be excited about it!

Template string queries

This version lets you take advantage of one of the biggest innovation in Python 3.14: the template strings, which allow you to write expressive and safe queries.

How does it look? Something like:

def fetch_person(conn, name):

    # 'name' will be handled safely: as a server-side parameter or
    # correctly quoted and escaped if client-side binding is required
    cur = conn.execute(t"SELECT * FROM people WHERE name = {name}")
    return cur.fetchone()

The syntax is the same as that of f-strings, introduced back in the venerable Python 3.6 (perhaps the feature that finally ended Python 2?), but now paired with the safety and adaptation flexibility of Psycopg 3. Template strings also help you generate dynamic SQL statements much more succinctly than with the psycopg.sql module:

def delete_something(conn, table_name, name):

    # Mixing client-side query composition with server-side parameters binding
    conn.execute(t"DELETE FROM {table_name:i} WHERE name = {name}")

    # Composing non-parametric statements entirely client-side
    conn.execute(t"NOTIFY {table_name + '.deleted':i}, {name:l}")

Check out the complete t-string support documentation for inspiration!

More flexible composite adaptation

Previously, it was only possible to adapt PostgreSQL composites to Python sequence types with a strict 1:1 mapping to the fields of the database type.

We have now gained extra flexibility: we can customize both how to create generic Python objects, for example ones only taking keyword arguments, and how to extract a sequence of fields from the attributes of non-sequence objects... Dataclasses anyone?

from dataclasses import dataclass
from psycopg.types.composite import CompositeInfo, register_composite

@dataclass
class MiniPerson:
    age: int
    name: str
    height: float | None = None

    @classmethod
    def from_db(cls, seq, info):
        return cls(name=seq[0], age=seq[1])

    def to_db(self, info):
        return [self.name, self.age]

conn.execute("CREATE TYPE mini_person AS (name text, age int)")
info = CompositeInfo.fetch(conn, "mini_person")

register_composite(
    info, conn, factory=MiniPerson,
    make_object=MiniPerson.from_db, make_sequence=MiniPerson.to_db)

conn.execute("SELECT ('John', 33)::mini_person").fetchone()[0]
# MiniPerson(age=33, name='John', height=None)

conn.execute(
    "SELECT (%(person)s).name || ' next year will be ' || (%(person)s).age + 1",
    {"person": MiniPerson(name="John", age=33)},
).fetchone()[0]
# 'John next year will be 34'

Solving the `fetchone()` annoyance with type checkers

If you use Mypy or other type checkers with Psycopg, you've probably seen false positives when calling fetchone(). Even if you are 100% certain your query will return a row, fetchone() is annotated as possibly returning None — so type checkers complain about patterns like:

cur.execute("SELECT count(*) FROM my_table")  # Always returns exactly one value
count = cur.fetchone()[0]   # Error: value of type "tuple | None" is not indexable

In Psycopg 3.3, the cursor has become an iterator, whereas it was previously only an iterable. The distinction is subtle but meaningful: an iterator holds its own iteration state and does not need to create a new object for each pass.

More importantly, this change means you can use next() or anext() to retrieve a record — and these functions never return None. This makes Mypy happy, and probably you too:

cur.execute("SELECT count(*) FROM my_table")
count = next(cur)[0]

Improvements to the connection pools

A connection pool’s parameters can now be changed dynamically — useful for example to support short-lived secret tokens as passwords, as requested by some cloud database providers.

A useful drain() method is now available to re-create all connections in a pool. This is helpful, for instance, when the database needs to be introspected to find the OIDs of extension types to register: without draining the pool the connections already in the pool would remain stale after the adapters have been configured.

...And more!

Other improvements include greater flexibility when navigating results after a fetchmany() call or after statements returning multiple result sets, the ability to reconfigure loaders after a query has run, and many other assorted enhancements. You can find the full list in the psycopg release notes and the pool release notes!

Your help is welcome

Psycopg is the de-facto standard for communication between Python and PostgreSQL — two major components powering countless businesses and mission-critical infrastructure. Maintaining such an important library to the highest standards of reliability, performance and security requires a lot of care and ongoing work.

If you use Python and PostgreSQL and want to help ensure that the interface between them remains robust and continues to improve, supporting new language and database features, please consider supporting the project 💜

Thank you very much, and happy hacking!

Automatic async to sync code conversion

2024-09-23T00:00:00Z

Psycopg 3 provides both a sync and an async Python interface: for each object used to perform I/O operations, such as Connection, Cursor, there is an async counterpart: AsyncConnection, AsyncCursor, with an intuitive interface: just add the right async or await keyword where needed:

# Familiar sync code
conn = psycopg.Connection.connect("")
cur = conn.execute("select now()")
print(cur.fetchone()[0])

# Similar async code
aconn = await psycopg.AsyncConnection.connect("")
acur = await aconn.execute("select now()")
print((await acur.fetchone())[0])

The decision to provide both sync and async code was made early in the development of Psycopg 3 and most of the internal code is written in a way to be compatible with both sync and async code, in order to keep code duplication to a minimum. This was achieved by making all the libpq communication async, and writing the network code as generators, yielding at the moment they need to wait, isolating the differences in the sync/async wait policy all in wait() functions.

This helped to minimise the async/sync differences in the code related to the communication between PostgreSQL and Psycopg. However, the interface between Psycopg and the Python user is still a lot to maintain, and consists of a lot of code that is very similar, almost duplicated, between the sync and async sided. Apart from the obvious async/await keywords, there would be subtle implementation differences, for example:

using asyncio functions instead of blocking counterparts, for instance await asyncio.sleep() instead of time.sleep();
asyncio.create_task(f(arg1, arg2)) is similar to thread.Thread(f, (arg1, arg2)).start();
threading.Event has a asyncio.Event counterpart whose lock() method doesn't have a timeout, parameter, so event.wait(timeout=10) needs to be rewritten as asyncio.wait_for(event.wait(), timeout=10).

Up until Psycopg 3.1, the two variants of each object were kept in sync manually. Every time changes were made on the sync side, they had to be ported to the async side, with cumbersome and noisy diffs, with subtle differences being introduced from time to time. Even the tests were pretty much duplicated (with some sync tests being accidentally lost on the async side, or vice versa). It seemed like a situation that could have been improved.

This is so boring that...

...a computer should do it for me instead.

Writing the async side starting from the sync side? Actually, the opposite. It is obvious that the async side has more information than the sync side (every method definition and call clearly indicates whether it will block or not) and most of the differences are minimal and repetitive. What we want then is a script that takes asyncio-based source code as input and outputs equivalent sync code.

This article describes what we did to implement such a script and how we used it for the initial transformation (replacing manually written sync code with auto-generated code without loss of quality) and how we are currently using it to maintain the Psycopg 3 codebase.

Abstract Syntax Tree

You would be tempted to write a bunch of regular expressions to just scrub away every async and await keyword found, but the source code is probably the wrong level to attack the problem: Python knows how to parse Python itself well and can allow us to reason at a higher level.

A better tool to work with is the Abstract Syntax Tree (AST): an in-memory representation of the code obtained after parsing. At this level we manipulate objects that represent "the for loop", or "the function call", and we are not fooled by unexpected spaces, extra brackets, comments, literal strings, and other traps.

The Python 'ast' module is the obvious starting point: if you have a bit of source code such as:

import asyncio

async def async_square(n):
    # Squares are slow
    await asyncio.sleep(1)

    return n * n

you can pass it to the module to see the AST tree that represents it:

$ python -m ast ast1.py
Module(
   body=[
      Import(
         names=[
            alias(name='asyncio')]),
      AsyncFunctionDef(
         name='async_square',
         args=arguments(
            args=[
               arg(arg='n')],
            defaults=[]),
         body=[
            Expr(
               value=Await(
                  value=Call(
                     func=Attribute(
                        value=Name(id='asyncio'),
                        attr='sleep'),
                     args=[
                        Constant(value=1)]))),
            Return(
               value=BinOp(
                  left=Name(id='n'),
                  op=Mult(),
                  right=Name(id='n')))],
         decorator_list=[],
         returns=Name(id='float'))])

You can see, highlighted, the nodes in the tree representing the main statements in the code: the tree represents a module, whose body contains two statements - an import and an async def - with the function body defining an await call and a return statement.

The same ast module can perform the reverse transformation, converting an AST tree back to source:

$ python -c "import ast; print(ast.unparse(ast.parse(open('ast1.py').read())))"
import asyncio

async def square(n: float) -> float:
    await asyncio.sleep(1)
    return n * n

As you can see, the transformation back to code is unfortunately not a perfect reconstruction of the original code, it is only equivalent, with missing comments and different spacing. This is because the syntax tree is abstract and whitespaces and comments don't affect it. If you wanted to take those details into account you would need a concrete syntax tree (something like that exists, but I haven't played with it).

Changing whitespaces is not a problem, but losing comments can be, especially when they are used to control linters (such as Flake8's noqa or Mypy's type: ignore), or simply when you happen to be a human being and want to read the source code. Fortunately there is a simple wrapper module, ast-comments, which does exactly what it says on the tin: it introduces Comment nodes as part of an AST. Playing around with it, it turned out to be a good compromise between an abstract and a concrete syntax tree, after some taming of the comments placement.

Du AST Mich

To perform the code transformation, we will walk over the abstract syntax tree and we will perform some operation to generate a different tree of our liking. Typically, this type of operation is performed using an implementation of the visitor pattern.

This pattern can be incredibly useful whenever you need to perform operations on data structures composed of heterogeneous nodes (I've seen it in applications ranging from converting UML representations to code, converting markup language to HTML, converting Kubernetes manifests to Helm charts, converting annotated lyrics files to Ukulele tab sheets...); unfortunately many of the descriptions of the pattern you can find online fail to make its brilliance immediately apparent (the Wikipedia page is pretty bad at it) because they have historically focused on solving the double-dispatch problem in static languages like as C++ or Java (which is trivial in a dynamic language like Python) rather than focusing on the awesome things you can do with it.

In a nutshell, you will have an object that traverses an input data structure, element by element, building an output structure in the process, allowing you to run different code and to perform different manipulations depending on the type of element being traversed.

In our case, both the input and the output are AST trees, which will happen to be very similar to each other (since we are just trying to translate some subtle differences from one Python module to another): for many nodes, the visitor will just output a copy of it (for example, the return statement in the above example is unchanged). But, if we see a pattern of interest, we can tell our visitor to produce a different node.

The ast module provides a base class ast.NodeTransformer which implements the node traversal and tree production parts. By itself it doesn't perform any operations on the nodes, so it just produces a copy of the input tree. However, by subclassing the class and adding visit methods, you can implement node-specific transformations.

With the AST node transformer, the method called is based on the name of the node being visited; for example, if you add a method called visit_Import to your subclass, the visitor will call it whenever it traverses an Import node, giving you the chance to manipulate an import statement. You can then decide whether you want to change some of the details of the node (drop some imports, change some names), or replace the node with something completely different (such as replacing an async function definition with a sync one).

Let's say that we want to produce a sync version of the above script: the differences should be the following:

@@ -1,7 +1,7 @@
-import asyncio
+import time

-async def async_square(n: float) -> float:
+def square(n: float) -> float:
     # Squares are slow
-    await asyncio.sleep(1)
+    time.sleep(1)

     return n * n

In our toy example, we want to convert the asyncio module into the time module (which is obviously not the right thing to do in the general case, but let's keep the example simple). The following script implements the transformation and prints the converted module:

import ast

class MyTransformer(ast.NodeTransformer):
    def visit_Import(self, node):
        for alias in node.names:
            if alias.name == "asyncio":
                alias.name = "time"

        return node

with open("ast1.py") as f:
    tree = ast.parse(f.read())
tree = MyTransformer().visit(tree)
print(ast.unparse(tree))

The script will print the new source, with an import time replacing the original import asyncio.

Changing the async call is a bit trickier: we want to change the highlighted parts of the original tree:

value=Await(   << this node must be dropped, replaced by its value
   value=Call(
      func=Attribute(
         value=Name(id='asyncio'),  << we want time here
         attr='sleep'),
      args=[
         Constant(value=1)],
      keywords=[]))),

Adding the following two methods to the above class will implement what has been described.

def visit_Await(self, node):
    new_node = node.value  # drop the node, continue to operate on the value
    self.visit(new_node)
    return new_node

def visit_Call(self, node):
    match node.func:
        case ast.Attribute(value=ast.Name(id="asyncio"), attr="sleep"):
            node.func.value.id = "time"
    return node

To make sense of how these methods operate on their input nodes, and then to implement your own transformations, you can always look at the output of python -m ast in order to see the attributes on each node and how they are nested.

The visit_Call method shows how the structural pattern matching introduced in Python 3.10 comes in handy for this operation. The method is called for each function call found in the input code; checking whether the one just received requires any manipulation would have involved a cascade of ifs (the value is a Name, its id is asyncio, the attr is sleep...) which becomes pretty ugly pretty quickly, whereas instead a match statement can describe a complex nested test very succinctly.

Problems with `sleep()`

Performing the transformation from asyncio.sleep to time.sleep for real is much more complex than this. What if our source includes from asycio import sleep, Event? We would have to split the import into several parts:

from time import sleep
from threading import Event

and the latter should be treated differently later because the two Event objects have a different wait() signatures.

To help with this operation, in Psycopg 3 we introduced an internal '_acompat' module (actually two, because the pool is released separately and uses different functions; actually three, because the tests also have their own...) to expose pairs of functions or objects that should be used alternatively in sync or in async mode.

For example we can solve the sleep() problem with:

# module _acompat.py

import time
import asyncio

sleep = time.sleep

def asleep(seconds: float) -> Coroutine[Any, Any, None]:
    """
    Equivalent to asyncio.sleep(), converted to time.sleep() by async_to_sync.
    """
    return asyncio.sleep(seconds)

Now it's easy to use from ._acompat import asleep; await asleep(1) and do some simple name substitutions in the AST: the resulting statement from ._acompat import sleep; sleep(1) will work as expected.

Other goodies we have implemented to help unify async and sync code are aspawn/spawn and agather/gather to unify threads and asyncio tasks creation, alist() to encapsulate [x for x in await iterable] in a way that can be easily converted to list(iterable) and many other helpers to smooth the transition.

When everything else fail

There may be parts of the codebase where the difference between sync and async versions is too difficult to handle in a practical way, and is not worth to put together a complex matching for a complex, one-off case. What we want is a simple "if async, do this, else do that".

We have solved this problem by using a pattern like:

if True:  # ASYNC
    foo()
else:
    bar()

The AST with this code, including the comments, looks like:

Module(
   body=[
      If(
         test=Constant(value=True),
         body=[
            Comment(value='# ASYNC', inline=True),
            Expr(
               value=Call(
                  func=Name(id='foo'),
                  args=[],
                  keywords=[]))],
         orelse=[
            Expr(
               value=Call(
                  func=Name(id='bar'),
                  args=[],
                  keywords=[]))])],
   type_ignores=[])

Our transformation will find the ASYNC comment: in this case it will simply discard the if side of the condition, as well as the if itself, and will leave only the else branch in the sync code, allowing you to discard unneeded imports or other code that would simply be invalid in the sync context.

This pattern is also efficient, because the Python compiler is able to recognise that if True will always take the first branch, so it will discard the test and the code in the else branch. The dis(assembler) module shows no jump and that no reference to the bar() function call:

$ python -m dis ast3.py
  1           0 NOP

  2           2 LOAD_NAME                0 (foo)
              4 CALL_FUNCTION            0
              6 POP_TOP
              8 LOAD_CONST               1 (None)
             10 RETURN_VALUE

Conversion methodology

Once we have our conversion script, how do we use it to actually convert the code base, making sure to not break it? The process, for us, was iterative: going module by module and adding features to the script until all the "duplicated" modules were complete.

For each module to be converted, the procedure was roughly as follows.

First step: refactoring the code with the intention of not changing any behaviour, but of making the async module as similar as possible to the sync module. This might have meant some code reorganisation, the renaming of some variables, the swapping of some function definitions, the rediscovery of some forgotten skeletons and a chance of giving them a proper burial.

Often we would have implemented some non-I/O related helper function on the sync side and imported it on the async side:

# connection.py

def clean_up_conninfo(conninfo):
    ...  # hack hack
    return better_conninfo

def connect(conninfo):
    better_conninfo = clean_up_conninfo(conninfo)
    conn = wait(connection_gen(bettern_conninfo))
    return conn

# connection_async.py

from .connection import clean_up_conninfo

async def connect_async(conninfo):
    better_conninfo = clean_up_conninfo(conninfo)
    aconn = await async_wait(connection_gen(bettern_conninfo))
    return aconn

In this case we would have moved the shared functionality in a separate internal module and imported the function on both the sides:

# _connection.py

def clean_up_conninfo(conninfo):
    ...  # hack hack
    return better_conninfo

# connection.py

from ._connection import clean_up_conninfo

def connect(conninfo):
    better_conninfo = clean_up_conninfo(conninfo)
    conn = wait(connection_gen(bettern_conninfo))
    return conn

# connection_async.py

from ._connection import clean_up_conninfo

async def connect_async(conninfo):
    better_conninfo = clean_up_conninfo(conninfo)
    aconn = await async_wait(connection_gen(bettern_conninfo))
    return aconn

Now that the two modules are more similar we can run the test suite to verify that the library still works and can commit the current state to git.

Second step: run an async -> sync conversion with the current version of the script. Even running a no-op script is useful: it produces changes that can be easily seen with git diff, suggesting which conversion feature is missing, or what further cleanup we could have done in the code to make the sync and async flavours more similar.

For example, a no-op script that just copies the async side to the sync side, would show up in git diff as:

@@ -1,6 +1,6 @@
 from ._connection import clean_up_conninfo

-def connect(conninfo):
+async def connect_async(conninfo):
     better_conninfo = clean_up_conninfo(conninfo)
-    conn = wait(connection_gen(bettern_conninfo))
-    return conn
+    aconn = await async_wait(connection_gen(bettern_conninfo))
+    return aconn

The first feature to add to the conversion script is to remove the async and await keywords. Run the conversion and diff again and you will see:

@@ -1,6 +1,6 @@
 from ._connection import clean_up_conninfo

-def connect(conninfo):
+def connect_async(conninfo):
     better_conninfo = clean_up_conninfo(conninfo)
-    conn = wait(connection_gen(bettern_conninfo))
-    return conn
+    aconn = async_wait(connection_gen(bettern_conninfo))
+    return aconn

The next step is some renaming. If connect() and connect_async() are public functions we don't want to change their names. The script should have a name mapping function suggesting to convert:

connect_async -> connect
wait_async -> wait

Implementing this renaming in the AST we would bring us to the diff:

@@ -2,5 +2,5 @@

 def connect(conninfo):
     better_conninfo = clean_up_conninfo(conninfo)
-    conn = wait(libpq.connect_async())
-    return conn
+    aconn = async(libpq.connect_async())
+    return aconn

We are getting there. This remaining aconnn/conn is actually a gratuitous difference: we can change the async side and call the local variable conn without losing readability and obviously without changing any behaviour.

Committing the change on the async side and re-running the conversion would show no more difference on the sync side. At this point we can commit the whole project (any remaining but acceptable change on the sync side, the new features added to the conversion script, new entries in the renaming mapping...), run the tests to verify that no regression has been introduced, and move on to the next module.

This operation, in Psycopg 3, started at commit 765f663f and can be seen in the git history as a parallel branch that was eventually merged in 8bb0f9bf. The diff --stat shows a whopping:

99 files changed, 9697 insertions(+), 8486 deletions(-)

which is obviously a monster changeset, but mostly consists of incremental refactorings, conversions, finding new ways to minimise differences. It could be an interesting ride if you have a project where you need to introduce a similar automatic conversion.

The final result

Here is the Psycopg 3 async to sync conversion script (as of the Psycopg 3.2 release). At the time of writing, It processes 27 files and automatically generates about the 25% of the codebase. Some of the features it boasts:

the AST transformations described above, including tricks like recursion into strings containing code to be transformed, such as Mypy annotations expressed as strings, adjusting the output and the comments to make the resulting unparsed code almost as good as the handwritten side;
it inserts non-essential whitespace, and runs black on the output, in order to make the resulting code as uniform as possible to the original and as good for humans to work with (to read, debug, diff, etc);
since different Python versions may generate different ASTs and different output code, it can run in a Docker container, whose image is created on the fly using as base the Python image of the reference version;

it adds a useful disclaimer to the top of the file:

# WARNING: this file is auto-generated by 'async_to_sync.py'
# from the original file 'connection_async.py'
# DO NOT CHANGE! Change the original file instead.

it has a "check" mode that runs in Github Action upon every commit, as part of the lint step, and will fail if it finds any files to convert that haven't been committed;
the check mode has its own check: if any script containing the above disclaimer is not included in the list of files to be converted, it will throw an error (because a converted file has not been added to the automatic conversion list);
the check of the check also has its own check! If no file with the disclaimer is found then it means that something is wrong... Maybe the disclaimer has been rewritten and the check doesn't work anymore;
it can run in parallel and only on the files that have changed. Almost as good as make (but for certain tasks it is useful to have all the input files at once, therefore, "better than make").

The code is specific to the Psycopg 3 codebase and formatting style, so it's probably not ready to be used as it is in other projects. But it is probably a good starting point to to your own conversion: change the list of files to process, the name mapping, and you should be good to start.

Hope this helps. Happy hacking!

Psycopg 3.2 released

2024-06-30T00:00:00Z

It was quite the ride! But we made it!

After almost two years, 846 commits, more than 700 new tests, more than 20000 changes in 310 files (I didn't even realise that there were 310 files in this project...) Psycopg 3.2 has been released!

This release brings a few new feature and hopefully no meaningful non-backward compatible change. The whole list of changes is available in the changelog; these are some of the major points explained.

Numpy scalars support

In many scientific applications, Numpy scalars are widely used, either by themselves or in conjunction with regular Python values. However there was no support for storing them to the database and a conversion to normal Python values was necessary. Starting from Psycopg 3.2 storing Numpy scalars is automatic and the operation efficient.

A natural extension would be to convert between Numpy and PostgreSQL arrays too. However there hasn't been much demand for the feature, therefore it's currently on the back burner but can be implemented if there is demand.

PostgreSQL parameters

Psycopg uses placeholders such as %s and %(name)s to pass values to queries. These formats are familiar to Python developers, but they are quite foreign in PostgreSQL environment, because, natively, PostgreSQL uses a number-based placeholder format (such as $1, $2...) Psycopg, internally, converts the first format into the second.

It is now possible to execute queries using the PostgreSQL format by using the raw query cursors, which should feel more familiar to PostgreSQL developers and maybe lower the barrier to convert programs using large bodies of native queries to Python (the PostgreSQL test suite, maybe?)

cur = psycopg.RawCursor(conn)
cur.execute("SELECT ($1 + $2) * $1", [3, 5]).fetchone()
(24,)

Scalar row factory

The example above shows a pretty common annoyance. How many times do you need a single value from the database and you are returned a tuple?

Psycopg normally emits records as Python tuples; the behaviour can be customized to return named tuples, dictionaries, or entirely custom objects with the use of row factories.

In the frequent case of a query returning a single value, the new scalar_row factory will return only that:

cur = psycopg.RawCursor(conn, row_factory=scalar_row)
cur.execute("SELECT ($1 + $2) * $1", [3, 5]).fetchone()
24

This is not a feature of RawCursor only, but it's independent from the choice of the cursor class. We just needed to fix the example above!

Libpq 17 features

In the upcoming PostgreSQL 17 release, the libpq (the PostgreSQL client library used internally by Psycopg) has seen an unusually intense activity, with the introduction of several new features.

Our friend Denis Laxalde has been quick to build features and improvements on top of these new functionalities. So, when Psycopg is used with libpq 17, it can benefit of features such as:

A new capabilities object can help to navigate the differences and to write programs either degrading gracefully or crashing helpfully if the libpq used doesn't offer a requested functionality.

Easier interaction with notifications

Psycopg 3 introduced a notifications generator to receive asynchronous notification from the database. However the generator turned out to be... difficult to stop! It could be stopped upon receiving a specific notification as a message, but, because of Python quirks, not easily from the rest of the program.

import psycopg
conn = psycopg.connect("", autocommit=True)
conn.execute("LISTEN mychan")
gen = conn.notifies()
for notify in gen:
    print(notify)
    # ehm... please kill me!

New timeout and stop_after parameters allow for better control of a notification listening task (often a component of larger applications) and to provide better ways to control its operations. Such as to kindly tell it that its services are not requested anymore without having to kill the whole program!

Less work for us!

An interesting internal change has helped us to reduce the amount of code to write and maintain.

All the Psycopg objects interacting with the network come in two flavours: one implementing "classic" blocking methods (with which concurrency in a process can be implemented via multi-threading) and one implementing asynchronous methods to participate in collaborative concurrency.

Thanks to an early design choice, all the libpq I/O interaction only happens via asynchronous functions and is shared by both the sync and the async objects; however the code implementing the outermost objects and highest level behaviour had to be pretty much almost duplicated, with the same features implemented almost identically with and without async/await keywords, bugs to be tested and fixed on two sides...

We have therefore developed an async_to_sync conversion tool to generate the synchronous code starting from the AST of the asynchronous counterpart. As a result, the 20-25% of the codebase is now automatically generated and doesn't require specific maintenance. The process of converting the sync side from hand-written to auto-generated has also highlighted subtle differences between async and sync behaviours, which have been addressed, and affects tests too.

The technique could be useful for other projects maintaining both sync and async code, and is interesting enough to require an article of its own to be written...

We need your help!

Psycopg, first v2, now v3, is the de-facto standard for the communication between Python and PostgreSQL, two major components of innumerable businesses and mission-critical infrastructures.

Maintaining such a critical library to the highest standard of reliability, completeness, performance requires a lot of care and work.

If you are a Python and PostgreSQL user and would like to make sure that the interface between the two is well maintained and continuously improved, please consider sponsoring the project and to be one of our sponsors 💜

Thank you very much, happy hacking!

Pipeline mode in Psycopg

2024-05-08T00:00:00Z

Version 3.1 of Psycopg added support for libpq pipeline mode, bringing significant performance boost, especially when network latency is important. In this article, we’ll briefly describe how it works from users’ perspective and under the hood while also providing a few implementation details.

Supporting libpq pipeline mode involved significant changes to the query processing logic in the driver. Yet, the challenge was to make it compatible with the “normal” query mode in order to keep the API almost unchanged and thus bring performance benefits to users without exposing the complexity of the batch query mode.

For the impatient, head out to the pipeline mode documentation of Psycopg: it’s self-consistent, explains nicely the details for client/server communication, as well as how things work from the user’s perspective.

Using the pipeline mode in Psycopg

Connection objects gained a pipeline() method to enable the pipeline mode through a context manager ("with" statement); so using it is as simple as:

conn = psycopg.connect()
with conn.pipeline():
   # do work

What is the pipeline mode for?

Postgres documentation contains advice on when the pipeline mode is useful. One particular case is when the application is doing many write operations (INSERT, UPDATE, DELETE).

For instance, let’s consider the following schema:

CREATE TABLE t (x numeric, d timestamp, p boolean)

and assume an application does a lot of queries like:

INSERT INTO t (x, d, p) VALUES (%s, %s, %s)

with distinct values. Maybe the application could make use of batch inserts such as executemany(), maybe not (e.g. because it needs to do some other operations between inserts, like querying another resource): this does not matter much.

Let’s put this together into a little demo.py Python program:

import math
import sys
from datetime import datetime
import psycopg

def create_table(conn: psycopg.Connection) -> None:
    conn.execute("DROP TABLE IF EXISTS t")
    conn.execute("CREATE UNLOGGED TABLE t (x numeric, d timestamp, p boolean)")

def do_insert(conn: psycopg.Connection, *, pipeline: bool, count: int = 1000) -> None:
    query = "INSERT INTO t (x, d, p) VALUES (%s, %s, %s)"
    for n in range(count):
        params = (math.factorial(n), datetime.now(), pipeline)
        conn.execute(query, params, prepare=True)

with psycopg.connect(autocommit=True) as conn:
    create_table(conn)
    if "--pipeline" in sys.argv:
        with conn.pipeline():
            do_insert(conn, pipeline=True)
    else:
        do_insert(conn, pipeline=False)
    row_count = conn.execute("select count(*) from t").fetchone()[0]
    print(f"→ {row_count} rows")

we’ll run our script as python demo.py [--pipeline], the --pipeline flag allowing to enable pipeline mode. Note that we passed prepare=True to Connection.execute(), in order to issue a PREPARE statement as we’ll emit the same query many times.

In general, each INSERT query will be fast to execute server-side. Without the pipeline mode enabled, the client will typically issue the query and then wait for its result (though it is unused here): thus the client/server round-trip time will probably be much larger than the execution time (on server). With the pipeline mode, we basically save these round-trips most of the times.

Interlude: tracing

When working on optimizing client/server communication, it’s essential to be able to monitor this communication at a reasonably low level. From Psycopg’s perspective, the boundary is the libpq. Fortunately, the library provides a tracing mechanism through the PQtrace function and friends.

The output of this function looks like (example taken from the PostgreSQL test suite):

F   68  Parse    "select_one" "SELECT $1, '42', $1::numeric, interval '1 sec'" 1 NNNN
F   16  Describe     S "select_one"
F   4   Sync
B   4   ParseComplete
B   10  ParameterDescription     1 NNNN
B   113 RowDescription   4 "?column?" NNNN 0 NNNN 4 -1 0 "?column?" NNNN 0 NNNN 65535 -1 0 "numeric" NNNN 0 NNNN 65535 -1 0 "interval" NNNN 0 NNNN 16 -1 0
B   5   ReadyForQuery    I
F   10  Query    "BEGIN"
B   10  CommandComplete  "BEGIN"
B   5   ReadyForQuery    T
F   43  Query    "DECLARE cursor_one CURSOR FOR SELECT 1"
B   19  CommandComplete  "DECLARE CURSOR"
B   5   ReadyForQuery    T
F   16  Describe     P "cursor_one"
F   4   Sync
B   33  RowDescription   1 "?column?" NNNN 0 NNNN 4 -1 0
B   5   ReadyForQuery    T
F   4   Terminate

Each row contains the “direction indicator” (F for messages from client to server or B for messages from server to client), the message length, the message type, and its content. This example shows messages from the Extended Query protocol.

In Psycopg, we have access to the low-level PGconn object, representing the libpq connection, through Connection.pgconn attribute.

Here’s how to enable tracing to stderr, for our demo.py program above:

from contextlib import contextmanager
from typing import Iterator
from psycopg import pq

@contextmanager
def trace_to_stderr(conn: psycopg.Connection) -> Iterator[None]:
    """Enable tracing of the client/server communication to STDERR."""
    conn.pgconn.trace(sys.stderr.fileno())
    conn.pgconn.set_trace_flags(pq.Trace.SUPPRESS_TIMESTAMPS | pq.Trace.REGRESS_MODE)
    try:
        yield
    finally:
        conn.pgconn.untrace()

def do_insert(conn: psycopg.Connection, *, pipeline: bool, count: int = 1000) -> None:
    # ...
    with trace_to_stderr(conn):
        for _ in range(count):
            conn.execute(query, params, prepare=True)

To pipeline or not to pipeline

If we run our demo script (without pipeline mode), we’ll typically get the following output:

F   69  Parse    "_pg3_0" "INSERT INTO t (x, d, p) VALUES ($1, $2, $3)" 3 NNNN NNNN NNNN
F   4   Sync
B   4   ParseComplete
B   5   ReadyForQuery    I
F   49  Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x00\x01' 8 '\x00\x02\xffffff8b\xffffff8fp~WN' 1 '\x00' 1 0
F   6   Describe     P ""
F   9   Execute  "" 0
F   4   Sync
B   4   BindComplete
B   4   NoData
B   15  CommandComplete  "INSERT 0 1"
B   5   ReadyForQuery    I
F   49  Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x00\x01' 8 '\x00\x02\xffffff8b\xffffff8fp~^\xffffff80' 1 '\x00' 1 0
F   6   Describe     P ""
F   9   Execute  "" 0
F   4   Sync
B   4   BindComplete
B   4   NoData
B   15  CommandComplete  "INSERT 0 1"
B   5   ReadyForQuery    I
[ ... and so forth ~1000 more times ... ]

we indeed see the client/server round-trips in the form of sequences of F messages followed by sequences of B messages for each query.

The first message sequence Parse+ParseComplete corresponds to the PREPARE statement. Next ones only have a Bind/Describe/Execute client messages followed by server response.

Now using the pipeline mode (run the script with --pipeline), we get the following trace:

F   69      Parse    "_pg3_0" "INSERT INTO t (x, d, p) VALUES ($1, $2, $3)" 3 NNNN NNNN NNNN
F   49      Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x00\x00' 8 '\x00\x02\xffffff8e<k\x0c\x16\xffffffe6' 1 '\x01' 1 0
F   6       Describe         P ""
F   9       Execute  "" 0
F   49      Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x00\x01' 8 '\x00\x02\xffffff8e<k\x0c\x18\xffffffcd' 1 '\x01' 1 0
F   6       Describe         P ""
F   9       Execute  "" 0
F   49      Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x00\x02' 8 '\x00\x02\xffffff8e<k\x0c\x19\xffffff8a' 1 '\x01' 1 0
F   6       Describe         P ""
F   9       Execute  "" 0
[ ... ~300 more of those ... ]
B   4       ParseComplete
B   4       BindComplete
B   4       NoData
B   15      CommandComplete  "INSERT 0 1"
B   4       BindComplete
B   4       NoData
B   15      CommandComplete  "INSERT 0 1"
B   4       BindComplete
B   4       NoData
B   15      CommandComplete  "INSERT 0 1"
[ ... ~300 more of those ... ]
F   49      Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x01<' 8 '\x00\x02\xffffff8e<k\x0c\xffffff96\xffffff8a' 1 '\x01' 1 0
F   6       Describe         P ""
F   9       Execute  "" 0
F   49      Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x01=' 8 '\x00\x02\xffffff8e<k\x0c\xffffff9c'' 1 '\x01' 1 0
F   6       Describe         P ""
F   9       Execute  "" 0
F   49      Bind     "" "_pg3_0" 3 1 1 1 3 2 '\x01>' 8 '\x00\x02\xffffff8e<k\x0c\xffffff9c\xffffff85' 1 '\x01' 1 0
F   6       Describe         P ""
F   9       Execute  "" 0
[ ... ]

We can see that the client sends more than 900 messages before the server replies (with the same number of messages). Clearly, this can have a huge impact on performance, especially when network latency matters. And indeed, this runs twice faster even though the Postgres server is on localhost!

What’s actually happening is that the client sends as many queries as possible, until the server decides it cannot manage more (in general because its output buffer is full, typically here because of the large integers we’re inserting), at which point the server sends back the results of all queries; rinse and repeat. Instead of producing small and frequent client/server round-trips, the pipeline mode optimizes network communication by producing large and scarce round-trips. The “downside” (remember we got a 2x speed-up) is that the client program needs to handle more data in memory in general.

How does it work?

As mentioned earlier, the entry point for the pipeline mode is the pipeline() method on Connection object which enters and exists pipeline mode. But what does this mean? Well, basically, this involves calling underlying PQ{enter,exit}PipelineMode functions.

But this does not tell much about how things work in Psycopg.

To actually understand how things work, we need to step back and read libpq pipeline mode documentation, in which section “Interleaving Result Processing and Query Dispatch” states:

The client application should generally maintain a queue of work remaining to be dispatched and a queue of work that has been dispatched but not yet had its results processed. When the socket is writable it should dispatch more work. When the socket is readable it should read results and process them, matching them up to the next entry in its corresponding results queue.

As often with PostgreSQL, everything is there although this paragraph is somehow enigmatic. However, it, in fact, describes the heart of the algorithm for the Psycopg driver (though it took us a while to grasp all the details implied by these few sentences…).

Socket communication

When the socket is writable it should dispatch more work. When the socket is readable it should read results […].

In Psycopg, socket communication for exchanging libpq messages is implemented through waiting functions and generators that are tied together by the I/O layer (either blocking or async): this is explained in details in Daniele’s blog post.

What’s important for the pipeline mode (mostly) is the generator part, as it is responsible for dispatching queries to or reading results from the socket. In contrast with normal query mode, where these steps are handled sequentially by independent logic, the pipeline mode needs interleaving result processing and query dispatch: this is implemented by the pipeline_communicate() generator. Without going too much into the details, we can notice that: - the function takes a queue of “commands”, e.g. pgconn.send_query_params() or similar, - it continuously waits for the socket to be either Read or Write ready (or both) (ready = yield Wait.RW), - when the socket is Read-ready (if ready & Ready.R:), results are fetched (calling pgconn.get_result()), - when the socket is Write-ready (if ready & Ready.W:), commands are sent (calling pgconn.flush() to flush the queue of previously sent commands, and then calling any pending one), - until the queue of commands gets empty.

Queueing work, processing results

Around the pipeline_communicate() generator described above, we need to handle the commands queue as well as the queue of results pending processing. The first part, filling the commands queue, is simply managed by stacking commands instead of directly calling them along with keeping a reference of the cursor used for execute(). The second part implies handling the output of pipeline_communicate() generator described above, a list of PGresult. Each fetched result item: - is possibly bound back to its respective cursor (the one where respective execute() originates from), - might trigger an error if its status is non-OK (e.g. FATAL_ERROR).

All this is handled in methods of the BasePipeline class (see methods prefixed with an _ at the end).

Integration with high-level features: transactions

Beside the low-level logic described above, implementing pipeline mode in Psycopg implied handling some Psycopg-specific features such as: transactions.

Transactions need special attention because of how error handling works in the pipeline mode. There is a few distinct cases that need to be handled properly, depending on whether the pipeline uses an implicit transaction or if it contains explicit transactions. But the general rule is that when an error occurs, the pipeline gets in aborted state meaning subsequent commands are skipped and prior statements might get persisted or not (depending on the usage of explicit transactions or not).

Consider the following statements, executed within a pipeline:

BEGIN;  # transaction 1
INSERT INTO s VALUES ('abc');
COMMIT;
BEGIN;  # transaction 2
INSERT INTO no_such_table VALUES ('x');
ROLLBACK;
BEGIN;  # transaction 3
INSERT INTO s VALUES ('xyz');
COMMIT;

SELECT * from s;
-> abc

The INSERT INTO no_such_table statement would produce an error, making the pipeline aborted; accordingly, the following explicit ROLLBACK will not be executed. And the next statements (“transaction 3”) will also be skipped.

Another example:

BEGIN;  # main transaction
INSERT INTO s VALUES ('abc');
BEGIN;  # sub-transaction
INSERT INTO no_such_table VALUES ('x');
ROLLBACK;
INSERT INTO s VALUES ('xyz');
COMMIT;

SELECT * from s;
-> []

Here, still due to the same error in INSERT INTO no_such_table, the final COMMIT statement is not executed and the main (outer) transaction is not committed (despite the inner sub-transaction is explicitly rolled back).

That’s typically something the user of a high level driver would not want.

In Psycopg, transactions are managed explicitly through the transaction() context manager method on Connection objects. So to preserve a consistent behaviour, its logic needed to be adapted for the pipeline mode. This got achieved by leveraging synchronization points through PQpipelineSync and nested pipelines.

Nested pipelines

In the libpq, there is no such thing as a nested pipeline as the connection can only enter pipeline mode once. What’s referred to as a “nested pipeline” in Psycopg is the operation to “isolate” a sequence of commands in a pipeline session through synchronization points. By doing so, we work around the surprising behaviour described above (where a committed transaction got rolled back). Here’s what happens:

with conn.pipeline():  # emits PQenterPipelineMode
  conn.execute(...)
  with conn.pipeline():  # emits PQpipelineSync
    conn.execute(...)
  # exiting the inner 'with' block emits PQpipelineSync
# exiting the outermost 'with' block emits PQexitPipelineMode

The PQpipelineSync operation resets the pipeline state, thus allowing subsequent commands to be run independently of whether previous ones succeeded or not. (It also triggers results to be sent back from the server, but that’s another matter.)

Pipelined transactions

By using nested pipelines for Psycopg transactions, we typically follow the “logical unit of work” pattern that’s mentioned in libpq pipeline mode documentation:

Pipelines should be scoped to logical units of work, usually (but not necessarily) one transaction per pipeline.

(Except that we’re not strictly use one pipeline per transaction, rather a nested one.)

In practice, it means that it’s safe to use with transaction(): block within a pipeline session as the semantics of both the transaction and the pipeline are preserved: the transaction either succeeds or fails overall, it only gets executed if previous commands in the pipeline session succeeded:

with conn.pipeline():
  conn.execute(...)
  try:
      with conn.transaction():  # implicit nested pipeline (with conn.pipeline())
          conn.execute(...)
  finally:
      # This will be executed independently of whether the previous
      # transaction succeeded or not.
      conn.execute(...)

So back to the (second) example above, if written using Psycopg:

>>> with psycopg.connect(autocommit=True):
...     with conn.pipeline():
...         with conn.transaction():
...             conn.execute("INSERT INTO s VALUES (%s)", ("abc",))
...             try:
...                 with conn.transaction():
...                     conn.execute("INSERT INTO no_such_table VALUES (%s)", ("x",))
...             except errors.UndefinedTable:
...                 pass
...             conn.execute("INSERT INTO s VALUES (%s)", ("xyz",))
...     conn.execute("SELECT * FROM s ).fetchall()
[('abc',), ('xyz',)]

we indeed get the inner transaction rolled back, and the outer one committed, just like without the pipeline mode.

That’s an implementation detail, the user does not need to know about this as the overall behaviour is hopefully natural.

Supporting libpq pipeline mode in Psycopg was a large milestone. It required months of work with a lot of thinking and testing. There is probably more to say about it, like how it transparently manages automatic prepared statement or how executemany() got optimized to use the pipeline mode implicitly (try adapting the demo script above to use it — hint: no need for the with pipeline: block). And be sure to read the Psycopg’s pipeline mode documentation soon!

This article, originally published at Pipeline mode in Psycopg, is used under CC BY-NC-SA (introduction shortened).

Psycopg 3.1 released

2022-08-30T00:00:00Z

Hello,

After several months of development, we are proud to release Psycopg 3.1!

Psycopg 3.1 is a gradual improvement on Psycopg 3.0, introducing new exciting features, redefining what can be done on the boundary between Python and PostgreSQL.

Pipeline mode

The pipeline mode is by far the biggest feature introduced in Psycopg 3.1, largely the work of Denis Laxalde and supported by Dalibo. In pipeline mode, Psycopg will send batches of commands to the server without waiting for a response for every operation, resulting in a massive speed improvement.

The pipeline mode is exposed to Python as a context block. Within the block, Psycopg will manage the pipeline in a transparent way, even allowing the use of features which break the pipeline flow, for instance to fetch results, or to manage transactions:

with conn.pipeline():
    cur = conn.cursor()
    for op in operations:
        cur.execute(op)

    with conn.transaction():
        cur.execute(op1)
        rec = cur.fetchone()
        cur.execute(op2, [rec.id])
    ...

Quantifying the speedup is difficult, as it depends on the network conditions and on the pattern of statements executed. In particularly bad conditions (250 ms of ping time between client and server), we measured a loop of 100 inserts to take 25 s in normal mode and just 0.5 s in pipeline mode, for a 50x speedup. Testing the same operation on a localhost connection, we have measured a >20x speedup (a 5000 inserts batch taking 0.3 s instead of 6.6 s). If the program requires results from the server before sending further statement (for instance to insert in a table and then use the new record's primary key to insert related records in different tables) you can expect less dramatic speedups.

Client-binding cursors

Psycopg 3 uses server-side binding, passing the query and adapted arguments separately. This allows to use several features otherwise unavailable, such as prepared statements. However, many types of statements, especially data-definition, don't support server-side parameters. The 'sql' module allows to compose statements on the client, but using it might require many changes to programs making heavy use of data-definition statements.

Psycopg 3.1 introduces a ClientCursor object, which makes psycopg2 programs easier to port. This cursor reproduces psycopg2's way of composing queries, which is not the most efficient, but for certain programs it is exactly what is needed.

with psycopg.connect(..., cursor_factory=ClientCursor) as conn:
    cur = conn.cursor()  # this is a client-side binding cursor
    # This statement doesn't support server-side parameters
    cur.execute("ALTER TABLE x ALTER y SET DEFAULT %s", ["hello"])

As a bonus, the client cursor reintroduces the handy and often requested mogrify() method, which returns the query merged with the parameters the way it is passed to the server:

cur.mogrify("ALTER TABLE x ALTER y SET DEFAULT %s", ["hell'o"])
"ALTER TABLE x ALTER y SET DEFAULT 'hell''o'"

Enum adaptation

Python has enums, PostgreSQL has enums... Why not map them into each other? Well for a start because they are actually pretty different from each other (Python enums have a type and value, Postgres ones are just identities) and because often, in programs, differences between the enums in the db and the code creep in.

Psycopg 3.1 introduces a flexible adapter between Python and Postgres enums. It can be used in a simple way when there is a one-to-one mapping between the enums:

=# CREATE TYPE numbers AS ENUM ('ONE', 'TWO', 'THREE');

class Numbers(Enum):
    ONE = auto()
    TWO = auto()
    THREE = auto()

info = EnumInfo.fetch(conn, "numbers")
register_enum(info, conn, Numbers)

conn.execute("SELECT 'TWO'::numbers").fetchone()[0]
<Numbers.TWO: 2>

conn.execute("SELECT pg_typeof(%s)", [Numbers.ONE]).fetchone()[0]
'numbers'

The facility can also be customized in order to adapt enums when the mapping is not one-to-one:

class NumbersPlus(Enum):
    ONE = auto()
    TWO = auto()
    THREE = auto()
    THREE_PLUS = auto()  # has some meaning in the program, but it is not stored

register_enum(info, conn, NumbersPlus,
    # - Items not mentioned map naturally
    # - This order gives THREE priority over THREE_PLUS when loading from db.
    mapping=[(NumbersPlus.THREE_PLUS, "THREE"), (NumbersPlus.THREE, "THREE")])

conn.execute("SELECT %s::text",
    [[NumbersPlus.ONE, NumbersPlus.THREE_PLUS]]
).fetchone()[0]
'{ONE,THREE}'

conn.execute("select '{TWO,THREE}'::numbers[]").fetchone()[0]
[<NumbersPlus.TWO: 2>, <NumbersPlus.THREE: 3>]

In partnership with CockroachDB

CockroachDB is a distributed database presenting an SQL interface on top of a distributed key-value store. Although it is a completely independent implementation, it uses the same PostgreSQL client-server protocol.

In the past few months, we have collaborated to create an even smoother integration, so that every PostgreSQL feature, also supported by CockroachDB, can be used in a transparent way: server-side cursors, pipeline mode, CockroachDB data types are all supported out-of-the-box.

CockroachDB also implements CHANGEFEED, a streaming query, which Psycopg can consume using its Cursor.stream() feature. This immediately receives every change happening in a database table, enabling interesting new ways to write distributed applications.

`executemany()` improvements

The executemany() method now supports returning the executed statements' output. You can now for instance execute an INSERT ... RETURNING on a batch of records and retrieve the ids associated to the newly inserted records.

executemany() automatically uses the pipeline mode already described, making use of pipeline mode speedups without changing any code in the programs using this method.

And many more improvements

The Two-phase commit protocol is now available as per DBAPI specification.
Asynchronous connections don't block on DNS names resolution on connect.
Cursor.copy() now takes parameters, like a normal query.
It is also possible to replace the writer of a 'Copy' object, in order to use psycopg just to format data in COPY format and to do something else with the data produced, for instance save it to a file for later processing.
...And many more improvements you can find in our release notes.

Thank you very much!

We hope you will enjoy to use Psycopg 3.1 and will benefit from its new features. Psycopg 3 is developed and maintained thanks to the support of our sponsors.

If you are a Python and PostgreSQL user and would like to make sure that the interface between the two is well maintained and continuously improved, please consider sponsoring the project 💜.

Thank you very much, happy hacking!

Psycopg 3.0 released

2021-10-13T00:00:00Z

Hello,

I am extremely excited to announce the first stable release of Psycopg 3!

Psycopg 3 is a complete rewrite based on the experience accumulated with the development and maintenance of psycopg2. Psycopg 3 targets all the current versions of Python (3.6-3.10) and PostgreSQL (10-14) and allows the use of modern Python development techniques, such as async and statically typed code. A list of the new features is available in the documentation.

This was a long journey: I would like to thank the people who have helped to make this project amazing with their ideas and their code: Denis Laxalde (row factories), Daniel Fortunov (transaction blocks), Jacopo Farina (PostGIS support) and many who have tested, helped, discussed, cheered for us.

And an immense thank you to the sponsors who have made this project possible: Postgres Professional and Command Prompt have been our biggest supporters so far, but many companies and individuals have given their generous contribution. Surely there will be more work to come in the future: if you want you can help sponsoring the project too.

If you would like to try out the project please check out the installation and usage instructions. We are eager to hear your feedback and to share this journey with you.

--- Daniele, on behalf of Psycopg

Psycopg 3.0 beta 1 released!

2021-08-30T00:00:00Z

We are immensely proud to release on PyPI the first beta package of Psycopg 3!

Psycopg 3 is a complete rewrite of Psycopg 2, maintaining the same fundamental libpq wrapper architecture and DB-API interface design, but exposing new features to better work with the newer versions of Python and PostgreSQL.

On the Python side, Psycopg 3 allows the use of asyncio-based concurrency and static typing. Many improvement to the Python interface make the library much simpler and more idiomatic to use,

On the PostgreSQL side, Psycopg 3 makes use of server-side parameters, prepared statements, binary parameters, and great support for COPY operations.

But the most outstanding feature of the project is not a technical one: Psycopg 3 was made possible by the great generosity of many sponsors, who have funded the development of the project. Among the many backers, we are especially grateful to Postgres Professional and Command Prompt, Inc, which have given the most outstanding support. But many other companies and individuals, each one in their capacity, have shown concrete support for free software development and progress. We sincerely hope that you will find this work useful and that you will feel proud for having contributed to it.

https://www.psycopg.org/sponsors/

Where do we go from here? The hope is that the interface of the adapter will not change excessively before a definitive 3.0 release: the project has already been used in a few production environments, in the past months, and a lot of real world feedback has already helped to improve the interface and functionalities. We invite you to test the project and give us your feedback. So...

pip install -U pip
pip install --pre psycopg[binary]

Please try it, test it, and let us know how it goes!

For more info you can dive into the docs: start from the install and usage pages, I'm sure you will find your way.

Happy hacking!

Building a Django driver for Psycopg 3

2021-08-02T00:00:00Z

One of the goals of the Psycopg 3 project is to make easy to port code developed from Psycopg 2. For this reason the creation of a Django backend (the module you specify in the settings as your database ENGINE) was a project with a double goal:

A Django driver is a way to make Psycopg 3 useful from the start, with the possibility of dropping it in a project transparently and have available, when needed the new features offered (for instance the superior COPY support).
The difficulty of introducing Psycopg 3 in the Django codebase and the type of changes required are indicative of the type of problems that could be found porting other projects.

...and it's done! A few days ago, the new Psycopg 3 Django backend could pass the entire Django test suite!

The implementation of the Django backend actually started several months ago, but it can be seen, from the test progression above, that its development had been suspended for several months. The problem, in the first attempts, was that too much of the Django code was in need of being adapted: this was a sign that the changes needed to use the new adapter were too invasive and that the same type of difficulties would have been met by everyone trying to replace Psycopg 2 with Psycopg 3. Back to the design board then, but hopefully the resulting adapter will behave mostly as you might expect and will not force users to change every query in their program (which would have been a deal breaker for most non-trivial projects).

The backend cannot be used with the current Django version: a few modifications to the Django codebase are needed in order to use it. These changes will be proposed to the Django project: if the Django maintainer will accept them, the driver should be usable starting from one of the next Django releases.

The aim of this article is to take a look at some of these modifications, to understand where the behaviour of Psycopg 3 diverges from its well known predecessor and how to work around the differences.

Server-side parameters binding

Many of these changes are the consequence of using server-side binding for the query parameters (using the libpq PQexecParams() function), instead of merging the arguments to the query on the client-side and using the simpler PQexec() function.

In the PQexec() case, the Postgres query parser has access to the literal values in the context where they are used and it looks like it is able to use this information in ways we don't appreciate...until we lose them. Do you think that text is the best PostgreSQL data type to convert Python strings to? I wish it was so simple. Below is an experiment with the psycopg.pq objects, the low level libpq wrapper that Psycopg 3 exposes:

>>> from psycopg.postgres import types
>>> conn.execute("create table testjson (id serial primary key, data jsonb)")
# <psycopg.Cursor [COMMAND_OK] [INTRANS] (database=piro) at 0x7f92d43d7b80>

# Note: $1, $2... are the low level Postgres placeholders.
# In a normal Psycopg query you would use classic '%s'.
>>> conn.pgconn.exec_params(
...     b"insert into testjson (data) values ($1)",
...     [b"{}"], [types["text"].oid])
# <psycopg_c.pq.PGresult [FATAL_ERROR] at 0x7f92cec70a90>

>>> print(_.error_message.decode("utf8"))
# ERROR:  column "data" is of type jsonb but expression is of type text
# LINE 1: insert into testjson (data) values ($1)
#                                             ^
# HINT:  You will need to rewrite or cast the expression.

Specifying the text Postgres type is an excessively strict type indication: in most cases, Postgres will not be able to automatically convert the value to the required type.

When we use a literal '{}' in the query, we are specifying an untyped literal. Postgres docs say that we can do the same using 0 as type OID for the parameter (see the paramTypes[] description). But it seems this isn't always the case. For instance:

>>> conn.execute("select concat(%s, %s)", ["foo", "bar"])
# ...becomes...
>>> conn.pgconn.exec_params(
...     b"select concat($1, $2)",
...     [b"foo", b"bar"], [0, 0])
# <psycopg_c.pq.PGresult [FATAL_ERROR] at 0x7f92d43db4d0>

>>> print(_.error_message.decode("utf8"))
# ERROR:  could not determine data type of parameter $1

This problem doesn't happen with every function: it seems to be only a problem with "variadic" functions, such as concat() or json_build_object(). As sporadic as it is, it doesn't seem like there is a universally correct way of mapping Python types to PostgreSQL type OIDs: we can try to get it right most of the time (so, by default, Psycopg 3 dumps Python strings using the OID 0), but places where this isn't right do exist...and they exist in Django, of course.

There are two different ways to work around the problem, both have their merit and one might be easier to use than the other in different contexts.

Add a cast to the placeholder: specifying %s::text (or other types) in your query, it is possible to disambiguate the type where "unknown" doesn't work:
```
>>> conn.execute("select concat(%s::text, %s::text)", ["foo", "bar"])
# ...becomes...
>>> conn.pgconn.exec_params(
...     b"select concat($1::text, $2::text)",
...     [b"foo", b"bar"], [0, 0])
# <psycopg_c.pq.PGresult [TUPLES_OK] at 0x7f92cebfb630>

>>> _.get_value(0, 0)
# b'foobar'
```
One place in Django where this was needed is in array comparisons, because they follow stricter rules than the base type comparisons and may require an explicit cast to work.

The other option is to use a different type than the Python str and handle it using a different dumper.

>>> conn.execute("select concat(%s, %s)", [Text("foo"), Text("bar")])
# ...becomes...
>>> conn.pgconn.exec_params(
...     b"select concat($1, $2)",
...     [b"foo", b"bar"], [types["text"].oid, types["text"].oid])
# <psycopg_c.pq.PGresult [TUPLES_OK] at 0x7f92cebfbbd0>

>>> _.get_value(0, 0)
# b'foobar'

This is proposed in Django and used, for instance, in the concat() case.

Note that both the solutions make the queries compatible with Psycopg 2 and 3: the %s::text casts are no problem in psycopg2 queries and psycopg2 is smart enough to notice that Text is a str subclass and to apply the same conversion rules used for the vanilla strings.

Server-side binding doesn't work for all the queries

Argument binding on the server only works for queries that select and modify data, but it doesn't work in the Data Definition Language. For example:

>>> conn.execute("create table test (id int default %s)", [42])
# Traceback (most recent call last):
#     ...
# psycopg.errors.UndefinedParameter: there is no parameter $1
# LINE 1: create table test (id int default $1)
#                                           ^

The solution for this type of problem is to use the psycopg.sql module to explicitly generate a query client-side and send it to the server without parameters:

>>> conn.execute(
...     sql.SQL("create table test (id int default {})")
...     .format(sql.Literal(42))
... )

A similar module is available in psycopg2 too so it is easy to write code that works for both versions: it's just the import statement that needs to be changed.

GROUP BY/ORDER BY with parameters

Another unexpected problem manifested with tests failing with a message like "column name must appear in the GROUP BY clause or be used in an aggregate function". This type of error appeared in tests leveraging Django ORM aggregation features, when the aggregation keys contain parameters.

For example, if you want to count your people by grouping them by the first two letters of their name, you may use a query such as:

SELECT left(name, 2) AS prefix, count(*) AS number
FROM people
GROUP BY left(name, 2)
ORDER BY left(name, 2)

If the "2" is actually a parameter, Django ends up composing and executing a query like:

cursor.execute("""
    SELECT left(name, %s) AS prefix, count(*) AS number
    FROM people
    GROUP BY left(name, %s)
    ORDER BY left(name, %s)
    """,
    [2, 2, 2])

If composed by the client, this query presents no problem, because the server query parser can clearly see the same expression in the output column and in the group/order predicates. However, moving to use server-side parameters, the query would be transformed to:

SELECT left(name, $1) AS prefix, count(*) AS number
FROM people
GROUP BY left(name, $2)
ORDER BY left(name, $3)

This query could be executed only if the three parameters are the same, but at parsing time the server cannot make sure this will be the case, failing with the error above.

If this query was under someone's control, it could be easily rewritten using named parameters instead of positional ones. I would personally write:

cursor.execute("""
    SELECT left(name, %(len)s) AS prefix, count(*) AS number
    FROM people
    GROUP BY left(name, %(len)s)
    ORDER BY left(name, %(len)s)
    """,
    {"len": 2})

which is transformed in a query with a single $1 placeholder used three times, and parsed successfully. Unfortunately Django only uses positional placeholders throughout its entire codebase and switching to a parameters mapping would be a very invasive change. A more localised change is to use column aliases: the same query can be rewritten as:

SELECT left(name, %s) AS prefix, count(*) AS number
FROM people
GROUP BY 1
ORDER BY 1

where "1" refers to the first output column. It is not a particularly loved syntax but it turns out useful here.

Probably not every database supports this syntax, so, in the proposed Django changeset, new feature parameters can be used to signal that the feature is accepted by specific databases, and currently enabled only for PostgreSQL.

A different package organisation

This is less mysterious to understand. The psycopg2 package has a bit of a chaotic organisation, with a couple of kitchen-sink modules (extensions and extras) containing a bit of everything: cursor subclasses, extra data types, utility functions, symbolic constants...starting from Psycopg 3, the package is organised in a more rational way and separated in different modules and sub-packages.

The bulk of the changes required to use Psycopg 3 from Django is just not to assume that talking to PostgreSQL will be done using psycopg2, but to import and use the objects according to the version of the driver in use.

Note that, with these modifications, Django is able to use both Psycopg 2 and 3, possibly both in the same project. While this is possible, it might not be the average use case in your project: more often than not you will be just interested in upgrading from Psycopg 2 to 3. In the simpler case of an update, you will just have to change your import statements, assuming unconditionally that the psycopg package is installed (there is no psycopg3 package: the same module name will do for the following major versions too) and just say goodbye to the glorious workhorse that psycopg2 has been so far. 👋

Takeaway points

In the design of Psycopg 3, a great effort has been made to allow a smooth adoption for users who have experience, and codebases, in psycopg2. The experience of the Django backend porting shows that most of the adjustments required fall in these categories:

Different package organisation
Over/under-specified types, which can be tweaked via casts and data wrappers
Inability to use server-side parameters, which can be worked around with client-side query composition

Hopefully now you know how to address these problems in case you are thinking of using Psycopg 3 in your next or current project!

Psycopg 2.9 released

2021-06-16T00:00:00Z

Psycopg 2.9 has been released!

This is a relatively small release compared to previous major releases. However the creation of the packages took a lot of effort. The previously used CI system now has reduced support for free software projects - it was decided that package building should be moved to GitHub Actions.

Packaging has also become more complex because of the evolution of the Python packaging standards and the need to support multiple architectures (Intel, ARM, PPC...).

Maintaining a project such as Psycopg requires a lot of effort. For this reason, we are extremely grateful to all our sponsors who are enabling the maintenance and development of Psycopg. Thank you very much!

What’s new in psycopg 2.9

with connection starts a transaction on autocommit transactions too (ticket #941).
Timezones with fractional minutes are supported on Python 3.7 and following (ticket #1272).
Escape table and column names in copy_from() and copy_to().
Connection exceptions with sqlstate 08XXX reclassified as OperationalError (a subclass of the previously used DatabaseError) (ticket #1148).
Include library dirs required from libpq to work around MacOS build problems (ticket #1200).

Other changes:

Dropped support for Python 2.7, 3.4, 3.5 (ticket #1198, ticket #1000, ticket #1197).
Dropped support for mx.DateTime.
Use datetime.timezone objects by default in datetime objects instead of FixedOffsetTimezone.
The psycopg2.tz module is deprecated and scheduled to be dropped in the next major release.
Provide PEP 599 wheels packages (manylinux2014 tag) for i686 and x86_64 platforms.
Provide PEP 600 wheels packages (manylinux_2_24 tag) for aarch64 and ppc64le platforms.
Wheel package compiled against OpenSSL 1.1.1k and PostgreSQL 13.3.
Build system for Linux/MacOS binary packages moved to GitHub Actions.

Designing a connection pool for psycopg3

2021-01-17T00:00:00Z

The psycopg2 pool is a pretty simple object, little more than... a pool of open connections, and I think it falls short in several ways:

the top usability problem is the fact that it cannot be used as context manager;
if a connection is broken it is not noticed it until it is used by a client;
if minconn connections are already taken, a new one is created and disposed of as soon as finished using, regardless of whether other clients may need it;
if more than maxconn connections are requested the client will receive an error.

For psycopg3 I would like something better. I have read around, looking into other pool implementations to figure out what a well designed connection pool ought to do (a very well thought one seems the Java HikariCP) and these are a few ideas I'd like to work on: they are here for a feedback, before I jump into enthusiastically implementing the wrong thing...

Client interface

Context manager

In modern Python it is expected that resources are handled by a context manager, so the canonical way to use a pooled connection should be:

with pool.connection() as conn:
    cur = conn.cursor()
    cur.execute("...")
    consume_data(cur.fetchall())

Note that, because there is a Connection.execute() method, the minimal use would be:

with pool.connection() as conn:
    conn.execute(DML_QUERY)

Blocking behaviour

Another important usability point is the behaviour in the face of an exhausted pool: the sane behaviour here is to block: not forever, but until either a connection is available or until a timeout expires, and only then throw an exception, so that in the face of a spike there is a buffer before requests start to get killed. I would expect no crash here:

async def worker(conn, work_for_sec):
    await conn.execute("select pg_sleep(%s)", (work_for_sec,))

pool = psycopg3.AsyncPool(maxconn=4, connection_timeout_sec=1.0):
for i in range(8):
    async with pool.connection():
        create_task(worker(conn, work_for_sec=0.5))

Which also illustrates that, as many other objects in psycopg3, there would be an asyncio version of the object and a sync one (which would work with normal threads and with Eventlet/gevent green threads).

Connection configuration

There should be a way to configure a Python connection, for instance to register adapters, before it is available to the application. Is subclassing the best way to do it? Someone might find more useful to choose dynamically what configuration to use.

# Subclassing:

class MyPool(psycopg3.pool):
    def configure(self, conn):
        MyAdapter.register(conn)
        return conn

pool = MyPool()

# Configuring:

def my_configure(self, conn):
    MyAdapter.register(conn)
    return conn

pool = psycopg3.Pool(configure_func=my_configure)

Exclusive connections?

Should there be a way to create an exclusive connection? Sometimes I need one, in an otherwise pooled application, e.g. to receive notifications:

def listen_notifications():
    with pool.exclusive() as conn:
        conn.autocommit = True
        for notify in conn.cursor().notifies():
            handle_notification(notify)

But I don't think there is such need if the connection parameters are made available on the pool:

pool = Pool(maxconn=config.pool_maxconn, args=(config.db_dsn,))

def listen_notifications():
    with psycopg3.connect(*pool.args, **pool.kwargs):
        conn.autocommit = True
        for notify in conn.cursor().notifies():
            handle_notification(notify)

which leaves to the user the choice between what connection class to use, how to configure it, etc. So that's probably not so useful.

Internal behaviour

Pool worker

If we desire the pool to be a fast provider of connection, not slowing down the application operations, certain things should happen behind the scene:

do we have a spike, so we need more connections?
do we have a moment of calm, so we can do with less connections?
are the connections in the pool still sane?

A trivial behaviour would be something like "if there are no connections available create a new one":

class Pool:
    def getconn(self):
        conn = self._get_a_connection_from_the_pool()
        if conn:
            return conn
        else:
            conn = psycopg3.connect(*self.args, **self.kwargs)
            self._put_it_in_the_pool_when_done(conn)
            return conn

This interesting article shows that this might not be the best way to do it, especially if the connection time is particularly long compared to the processing time. Growing the pool could be demanded to a background worker, following a strategy like:

class Pool:
    def getconn(self):
        if not self._pool():
            self.worker.create_a_new_connection_and_put_it_in_the_pool()
        return self.wait_for_a_connection_from_the_pool()

If there is a workers infrastructure in place there can me more jobs for them: periodically checking for the state of the unused connections in the pool, for instance, or closing them if they have been alive more than a configured amount of time (30 minutes?) and replacing them with a fresh one, (or not replacing them in case we have more than minconn connections and a previous spike has now passed).

What to do in case of connection failure?

The pool acts as a shield helping the application to reconnect in case the connection with the database is lost. However, when does this behaviour stop being sane? If the database is unreachable - maybe misconfigured, maybe moved - there is a risk that the application doesn't die, because the pool keeps insisting to reconnect, but it doesn't work either, because there is no working database connection.

In my experience with live systems this can be a difficult condition to diagnose: several times I have wished that an application, screaming loud on a dashboard, instead of trying to stay up complaining quietly in some logging stream. On the other hand maybe the state the application holds is precious enough that it can be worth to wait a few minutes before crashing and burning.

A first step towards a sane behaviour could be to die early, on startup, if the connection is not working when the first pool population is done: the program would fail hard and fast if the database - assumed to be a prerequisite - is nowhere to be found.

What if the database is missing in action during the program lifetime? Following connection attempts may be repeated, with an exponential backoff, until dying after a few minutes of fruitless attempts (with a sys.exit(1) or some other termination function which might be subclassed).

Connections usage pattern

In which order should the connections in the pool be used? As a stack (put back a connection on top of the stack, it will be the next one to use)? As a queue (put back the connection at the bottom of the queue, try to use them uniformly)? Randomly (whatever comes out from whatever hash map used to keep the connection)?

Is there a reason to prefer one way or the other? ISTM that a stack behaviour allows a better reuse of prepared statements and an easy implementation of the max_idle_sec parameter (get connections from the top of the stack, evict idle ones from the bottom). I am sure I haven't thought about this well enough though.

Proposed API

Given the behaviour described, the methods available on the pool object might look like the ones described here.

The interface would be implemented in two version: a Pool class using normal blocking functions and threads for concurrency (which greenlet libraries such as Eventlet or gevent would be able to monkeypatch) and an AsyncPool implementation using the usual await and async with on blocking methods and using asyncio.Task for concurrency.

connection(timeout_sec=None) method:

Open a context block and return a connection from the pool. The connection is returned to the pool at the end of the block.

On block exit, if a transaction is open, commit or roll back an open transaction, according to whether an exception has been raised in the block, consistently with what the connection block does.

getconn(timeout_sec), putconn(conn) methods:

Obtain a connection from the pool and return it. To use if, for some reason, the context manager cannot be used.

On putconn() check the state of the connection: if it is broken dispose of it and create a new one; if in state of transaction of error roll back the transaction, consistently with what Connection.close() does.

close()

Close all the connections in the pool. Further attempts to create a new connection will fail immediately with a helpful message (like, "dude, what do you want to do, seriously?")

configure(conn):

Configure a connection before making it available to the pool. The default implementation is no-op: subclasses may override it to configure the connections; alternatively a configure_func might be passed to the Pool constructor.

get_info():

Return stats about the behaviour of the pool so far (connections open, reused, returned...) for monitoring.

terminate()

Terminate the program, invoked by a pool worker after connection attempts have been repeatedly fruitless. It is exposed to allow overriding the default behaviour (sys.exit(1)) in a subclass.

get_maintenance_task():

Get the next maintenance task to perform on the pool. Having them available externally would make possible to have full control of when such tasks are executed (useful for testing or to provide some sort of sync behaviour, using no background worker).

Experimental idea: will drop it if it would make the pool implementation more difficult than it should be.

Key configuration parameters

These parameter would be passed to the class constructor.

minconn:: Minimum number of connections to keep open in the pool. If some are closed the pool should try to create new ones as quickly as possible to replace them. Proposed default: 4 (very defensive: to enable the pooling behaviour but to avoid to saturate a server unless configured up).
maxconn:: Maximum number of connections to open at any given time. If maxconn == minconn no extra connection is created in case more client requests arrive: they will just wait until a connection is made available again. If maxconn > minconn the pool can create new connections if the demand increases; later the extra connections created may be closed if deemed no more required (default: minconn).
args, kwargs:: Arguments to pass to the connection_factory to create a new connection (default: empty, connect as per PG* env vars, like psycopg3.connect()).
connection_factory:: The connection class to create (default: Connection for Pool, AsyncConnection for AsyncPool).
configure_function:: A function to configure the connection after its creation and before making it available to the pool. Alternative to subclassing the Pool class to configure Pool.configure(). It can be a callable taking a connection as parameter, or the dotted name of such callable, so that e.g. the function name might be passed to the application as an env var.
timeout_sec:: Default timeout before raising an exception if a connection cannot be served. It may be overridden by Pool.connection(timeout_sec=...) (default: 30 sec).
max_lifetime_sec:: Maximum time a connection should be kept in the pool and used. After such time, when the connection is not in use, it can be closed and replaced by a new one (default: 30 minutes - 10% random factor to avoid mass evictions).
max_idle_sec:: Time a connection can sit idle in the pool before being removed. Only connections above minconn are removed, if maxconn allows to create them (default: 10 min + 10% random).
terminate_after_sec:: Number of seconds to wait for a reconnection attempt before giving up and terminate the program, calling Pool.terminate() (default: 5 min, starting from 1sec ± 10%, doubled up until the precise threshold, then kaputt).
num_workers:: Number of background workers to perform maintenance tasks. If set to 0 the dynamic characteristics of the pool might be downgraded (e.g. creating a new connection will happen in the requesting thread, blocking it for the time it takes). Background jobs might be executed by the application calling get_maintenance_task() (default: 3).

Thoughts?

Please let me know what you think your Best Ever Connection Pool for psycopg3 should do. Thank you very much!

The psycopg3 adaptation system

2020-11-24T00:00:00Z

The adaptation system between Python objects and PostgreSQL types is at the core of psycopg2 and psycopg3. The flexibility of the psycopg2 adaptation system provides good out-of-the-box object mapping and allows users to customise it to suit any need. Do you want your decimal numbers returned as float because you need speed over pennies? Do you want to map PostgreSQL Infinity dates to the 25th of December 3099? That's certainly doable.

The psycopg3 adaptation system needs some modification compared to psycopg2, because psycopg3 uses the "extended query protocol" to send query parameters separately from the query. Together, with the differences to accommodate, there is also a chance to improve a system that has been in use for several years and has shown its shortcomings together with its strengths.

Server-side binding

Server-side parameter binding has been a long-time desired feature. So far psycopg2 has adapted arguments on the client-side:

# psycopg2
cursor.execute(
    "INSERT INTO tbl (s, n, d) VALUES (%s, %s, %s)",
    [42, "Hel'lo", date(2020, 12, 31)])

# has arguments adapted and quoted:
args[0] = "42"
args[1] = "'Hel''lo'"
args[2] = "'2020-12-31'::date"

# merged to the query:
query = query % args

# and passed as a single string:
libpq.PQexec(
    "INSERT INTO tbl (s, n, d) VALUES (42, 'Hel''lo', '2020-12-31'::date)")

The psycopg3 end user interface is unchanged, but behind the scenes will only perform adaptation, not quoting, and will send the arguments separately:

# psycopg3
cursor.execute(
    "INSERT INTO tbl (s, n, d) VALUES (%s, %s, %s)",
    [42, "Hel'lo", date(2020, 12, 31)])

# has arguments adapted:
args[0] = "42"
args[1] = "Hel'lo"
args[2] = "2020-12-31"

# and passed as separate information:
libpq.PQexecParams(
    "INSERT INTO tbl (s, n, d) VALUES ($1, $2, $3)",  # Postgres placeholders
    ["42", "Hel'lo", "2020-12-31"],                   # Postgres formats, no quoting
    oids=[INT8_OID, 0, DATE_OID])                     # Type indications if available

Server-side binding brings better performance, the possibility to use prepared statements and binary data, as well as better integration with server-side logging and monitoring. Theoretically it also brings better safety against SQL injections, but psycopg2 already does a good job at providing a safe path for parameter binding: in psycopg2, creating an unsafe query is already harder than doing things the right way.

However server-side binding brings a few incompatibilities, such as:

cannot send more than one query at once if parameters are used:

# Must use separate queries or psycopg3.sql client-side adaptation
cur.execute(
    "INSERT INTO tbl1 VALUES (%s); INSERT INTO tbl2 VALUES (%s)",
    [10, 20])

cannot use certain commands such as SET or NOTIFY with parameters:

# Must use "SELECT set_config('timezone', %s)"
cur.execute("SET timezone to %s", [timezone])

# Must use "SELECT pg_notify('channel', %s)"
cur.execute("NOTIFY channel, %s", [message])

cannot use the IN (...) construct:

# Must use "AND nation = any (%s)"
cur.execute("... AND nation in %s", [("IT", "FR", "DE")])

All in all, most queries will just work, however the few incompatibilities require a non-backward-compatible change in version.

A new adaptation system

Embracing the new type of communication requires to change the way Python parameters and PostgreSQL data types are adapted. The new system:

cannot use the whole SQL syntax, but must limit to literals. For instance the Python list ["foo, bar", "Hel'lo"] cannot be expressed with ARRAY['foo, bar', 'Hel''lo'] but must use array literal rules and become {"foo, bar",Hel'lo}.
Makes possible to specify type OIDs, in case that's useful (not always an easy choice).
Allows for the use of binary types, which is especially useful for a large binary blob, otherwise things become bloated by binary escaping and have to traverse the many layers of lexing/parsing in the server, each with its own memory copy. This can be done by simply using the %b placeholder over %s:
```
with open(image_name, "rb") as f:
    cur.execute(
        "INSERT INTO images (name, data) VALUES (%s, %b)",
        [image_name, f.read()])
```

It's also a good chance to review the work that must be done by the client to adapt values. psycopg2 creates several instances of "adapter" wrappers, one for each value adapted. In psycopg3, the adaptation objects have a different life cycle: choices based on the environment, such as the connection encoding, can be made once for each Python type, rather than once per value, doing radically less work for each converted object. You can check out the psycopg3 adaptation documentation for all the details.

Customising psycopg3 adaptation

Customising types adaptation can now be done using Dumper classes, either creating new ones or mapping existing ones on different Python classes to save to the database. For instance, the builtin DateDumper converts Python dates to PostgreSQL ones. PostgreSQL can handle an "infinity" date, which Python cannot. If we wanted to store Python's date.max to PostgreSQL infinity, we could create a subclass for the dumper and register it in the scope we want to use it, globally or just on a connection or cursor:

class InfDateDumper(DateDumper):
    def dump(self, obj):
        if obj == date.max:
            return b"infinity"
        else:
            return super().dump(obj)

cur.execute("SELECT %s::text, %s::text", [date(2020, 12, 31), date.max]).fetchone()
# ('2020-12-31', '9999-12-31')

InfDateDumper.register(date, cur)

cur.execute("SELECT %s::text, %s::text", [date(2020, 12, 31), date.max]).fetchone()
# ('2020-12-31', 'infinity')

The system is pretty symmetric and employs similar Loader objects to map OIDs to the code responsible for its decoding. For instance, if we wanted to reverse the above customisation and map PostgreSQL infinity date to date.max (instead of raising an exception), it could be done using a subclass of the builtin loader (or using an entirely new object if required):

class InfDateLoader(DateLoader):
    def load(self, data):
        if data == b"infinity":
            return date.max
        else:
            return super().load(data)

cur.execute("select '2020-12-31'::date, 'infinity'::date").fetchone()
# Raises DataError: Python date doesn't support years after 9999: got infinity

from psycopg3.oids import builtins
InfDateLoader.register(builtins["date"].oid, cur)

cur.execute("select '2020-12-31'::date, 'infinity'::date").fetchone()
(datetime.date(2020, 12, 31), datetime.date(9999, 12, 31))

The customisation automatically applies to recursive types, such as arrays or composite types: if the date loader is customised then the date array works as expected:

cur.execute("select '{2020-12-31,infinity}'::date[]").fetchone()
([datetime.date(2020, 12, 31), datetime.date(9999, 12, 31)],)

All in all the new adaptation system provides better performance and easier customisation compared to psycopg2. The new adapters are easier to compose, such as using them in COPY operations. And if client-side adaptation is still needed (to generate dynamically data definition statements, to prepare offline update scripts...) psycopg3.sql still allows for the flexibility of client-side query composition.

Project sponsorship

The new COPY support in psycopg3

2020-11-15T00:00:00Z

psycopg2 allows interaction with PostgreSQL COPY commands. However what is possible to do with them is relatively limited: the only possible interaction is with file-like objects:

there is no adaptation from Python objects to PostgreSQL, as there is for normal queries: data must be formatted "manually" by the user;
psycopg2 "pulls" data from the file: writing a system that produces data and pushes it into PostgreSQL is a very contrived operation, requiring to write a blocking file-like object;
there is no support for asynchronous copy.

psycopg3 addresses these shortcomings and makes it easy to write Python programs producing data and pushing it efficiently to the database using the COPY protocol.

psycopg2 adaptation system is designed to compose queries client-side, so it is concerned with the right use of the quotes: the python string O'Reilly is converted to 'O''Reilly', and the date(2020, 11, 15) to '2020-11-15'::date. These extra quotes get in the way of COPY, and there isn't an intermediate level where a conversion to string is performed, but no quote or other SQL construct are added.

psycopg3 uses the PostgreSQL extended query protocol and sends query and parameters separately. Parameters require adaptation to the PostgreSQL formats, but quoting, and quotes escaping, are no more its concern: the string O'Reilly doesn't need further manipulation and the date is converted only to the string 2020-11-15; types information are passed as additional separate information according to the libpq API.

The server-side format of these values are exactly what the COPY FROM command expects, so it's now easy to compose a row by adapting Python objects and to pass

O'Reilly\t2020-11-15\n

to the server. The mechanism to do so is exposed to Python by a new context manager, returned by the Cursor.copy() method, which enables to write:

with cursor.copy("COPY mytable FROM STDIN") as copy:
    copy.write_row(("O'Reilly", date(2020, 11, 15)))

Any list of tuples of values, or generator of sequences of values, can be used to push data into Postgres:

with cursor.copy("COPY mytable FROM STDIN") as copy:
    for record in my_generator():
        copy.write_row(record)

The copy operation is concluded as soon as the with block is exited and, in case a Python exception is raised, the error is pushed to the server, which will cancel the COPY operation in progress.

Binary format

The Copy object is also able to write data in binary format: at Python level this is entirely transparent:

with cursor.copy("COPY mytable FROM STDIN (FORMAT BINARY)") as copy:
    for record in generator:
        copy.write_row(record)

which might be more efficient than the textual format, but requires more care with the data types, as the server will not even perform an innocent int4 -> int8 cast for you.

Block-level COPY

psycopg2 allows (only) to operate on a COPY stream using a Python file-like objects: behind the scenes it reads one block of data from the source and writes it to the destination:

# From a file to the database
with open("input.data") as f:
    cursor2.copy_expert(f, "COPY mytable FROM STDIN")

# From the database to a file
with open("output.data", "w") as f:
    cursor2.copy_expert(f, "COPY mytable TO STDOUT")

This way of operating is not lost, but now the responsibility of moving data from one stream to the other is left to the user's code:

with open("input.data") as f:
    with cursor3.copy("COPY mytable FROM STDIN") as copy:
        while data := f.read(SIZE):
            copy.write(data)

with cursor3.copy("COPY mytable TO STDOUT") as copy:
    with open("output.data", "wb") as f:
        while data := copy.read()
            f.write(data)

While the new pattern is more verbose, it allows to produce and consume data with interfaces different than the file one, whereas previously it would have required to write some form of file-like adapter, blocking the copy in case no data was ready. This inversion of control allows, finally, the use of...

Asynchronous COPY

If your data producer, either at rows level or at blocks level, is capable of asynchronous operations, it is now possible to combine it asynchronously with COPY using exactly the same pattern as the sync code, only sprinkling the magic words here and there:

async with cursor.copy("COPY mytable FROM STDIN (FORMAT BINARY)") as copy:
    async for record in my_async_generator():
        await copy.write_row(record)

async with async_producer() as f:
    async with cursor.copy("COPY mytable FROM STDIN") as copy:
        while data := await f.read()
            await copy.write(data)

which covers an important use case pretty much impossible to introduce in psycopg2.

Project sponsorship

Psycopg 2.8.6 released

2020-09-06T00:00:00Z

Psycopg 2.8.6 has been released.

You can download the files from the new release from PyPI or install it with:

pip install --upgrade psycopg2

The changes included in the release are:

Fixed memory leak changing connection encoding to the current one (ticket #1101).
Fixed search of mxDateTime headers in virtualenvs (ticket #996).
Added missing values from errorcodes (ticket #1133).
cursor.query reports the query of the last COPY operation too (ticket #1141).
errorcodes map and errors classes updated to PostgreSQL 13.
Wheel package compiled against OpenSSL 1.1.1g.

Thank you very much to everyone who gave their contribution!

Psycopg 2.8.5 released

2020-04-06T00:00:00Z

Psycopg 2.8.5 has been released.

This release adds support for AIX and brings a few bug fixes.

You can download the files from the new release from PyPI or install it with:

pip install --upgrade psycopg2

The changes included in the release are:

Fixed use of connection_factory and cursor_factory together (ticket #1019).
Added support for logging.LoggerAdapter in LoggingConnection (ticket #1026).
Column objects in cursor.description can be sliced (ticket #1034).
Added AIX support (ticket #1061).
Fixed copy() of DictCursor rows (ticket #1073).

Happy hacking!

Psycopg 2.8.4 released

2019-10-20T00:00:00Z

Psycopg 2.8.4 has been released.

The release brings a few assorted bugfixes and adds support for Python 3.8 and PostgreSQL 12.

A more detailed changes list is

Fixed building with Python 3.8 (ticket #854).
Don't swallow keyboard interrupts on connect when a password is specified in the connection string (ticket #898).
Don't advance replication cursor when the message wasn't confirmed (ticket #940).
Fixed inclusion of time.h on linux (ticket #951).
Fixed int overflow for large values in Column.table_oid and Column.type_code (ticket #961).
errorcodes map and errors classes updated to PostgreSQL 12.
Wheel package compiled against OpenSSL 1.1.1d and PostgreSQL at least 11.4.

Psycopg 2.8.3 released

2019-06-14T00:00:00Z

We have released Psycopg 2.8.3, which includes a slight change to the logical replication.

Choosing the right frequency to send replication feedback messages from the client to the server was previously the developer's responsibility, with too many feedback messages being a waste of bandwidth and server resources, too few slowing down WAL cleanup and possibly preventing a server graceful shutdown.

Psycopg will now make sure that feedback is only sent after a certain period of time from the previous one, so that the client can simply call send_feedback() at each message without the fear of overwhelming the server.

Further details are available in the MR by Alexander Kukushkin and Oleksandr Shulgin. Thank you very much!

For completeness, the changes included in the release are:

Added interval_status parameter to start_replication() method and other facilities to send automatic replication keepalives at periodic intervals (ticket #913).
Fixed namedtuples caching introduced in 2.8 (ticket #928).

Psycopg 2.8.1, 2.8.2 released

2019-04-14T00:00:00Z

Hello,

We have just released Psycopg 2.8.2; a few days ago Psycopg 2.8.1 was released.

Some of the bugs addressed are ordinary teething problem with the 2.8 release, but an important change landed with 2.8.2: binary packages now ship with OpenSSL 1.1 instead of 1.0. This should fix concurrency problems on connection experienced both on Windows and Linux. Many thanks to Matthew Brett and Jason Erickson for this improvement!

The combined list of changes is:

Fixed "there's no async cursor" error polling a connection with no cursor (ticket #887).
Fixed RealDictCursor when there are repeated columns (ticket #884).
Fixed RealDictRow modifiability (ticket #886).
Binary packages built with OpenSSL 1.1.1b. Should fix concurrency problems (ticket #543, ticket #836).

Happy hacking!

Psycopg 2.8 released

2019-04-04T00:00:00Z

After about two years from the previous major release, psycopg 2.8 is finally here!

Among the highlights, PostgreSQL errors are now mapped to Python exceptions for a more idiomatic way to handle them. Several additions allow a better insight of the connection status and query results.

Behind the scene, asynchronous communication and concurrency received several improvements, and dropping support for older versions of Python gave the chance to refactor and modernise the codebase (with the especial help from Jon Dufresne who ruthlessly butchered our code into a streamlined pulp).

Thank you very much to everyone contributing so far. Happy hacking!

You can download as usual from the canonical urls:

Or just use pip:

pip install psycopg2

New features:

Added errors module. Every PostgreSQL error is converted into a specific exception class (ticket #682).
Added encrypt_password() function (ticket #576).
Added BYTES adapter to manage databases with mixed encodings on Python 3 (ticket #835).
Added table_oid and table_column attributes on cursor.description items (ticket #661).
Added connection.info object to retrieve various PostgreSQL connection information (ticket #726).
Added get_native_connection() to expose the raw PGconn structure to C extensions via Capsule (ticket #782).
Added pgconn_ptr and pgresult_ptr to expose raw C structures to Python and interact with libpq via ctypes (ticket #782).
sql.Identifier can represent qualified names in SQL composition (ticket #732).
Added ReplicationCursor.wal_end attribute (ticket #800).
Added fetch parameter to execute_values() function (ticket #813).
str() on Range produces a human-readable representation (ticket #773).
DictCursor and RealDictCursor rows maintain columns order (ticket #177).
Added severity_nonlocalized attribute on the Diagnostics object (ticket #783).
More efficient NamedTupleCursor (ticket #838).

Bug fixes:

Fixed connections occasionally broken by the unrelated use of the multiprocessing module (ticket #829).
Fixed async communication blocking if results are returned in different chunks, e.g. with notices interspersed to the results (ticket #856).
Fixed adaptation of numeric subclasses such as IntEnum (ticket #591).

Other changes:

Dropped support for Python 2.6, 3.2, 3.3.
Dropped psycopg1 module.
Dropped deprecated register_tstz_w_secs() (was previously a no-op).
Dropped deprecated PersistentConnectionPool. This pool class was mostly designed to interact with Zope. Use ZPsycopgDA.pool instead.
Binary packages no longer installed by default. The psycopg2-binary package must be used explicitly.
Dropped PSYCOPG_DISPLAY_SIZE build parameter.
Dropped support for mxDateTime as the default date and time adapter. mxDatetime support continues to be available as an alternative to Python's builtin datetime.
No longer use 2to3 during installation for Python 2 & 3 compatibility. All source files are now compatible with Python 2 & 3 as is.
The psycopg2.test package is no longer installed by python setup.py install.
Wheel package compiled against OpenSSL 1.0.2r and PostgreSQL 11.2 libpq.

Psycopg 2.7.5 released

2018-06-17T00:00:00Z

psycopg2 version 2.7.5 has been released, fixing a few bugs found in the last months:

Summary of changes:

Allow non-ascii chars in namedtuple fields (regression introduced fixing ticket #211).
Fixed adaptation of arrays of arrays of nulls (ticket #325).
Fixed building on Solaris 11 and derivatives such as SmartOS and illumos (ticket #677).
Maybe fixed building on MSYS2 (as reported in ticket #658).
Allow string subclasses in connection and other places (ticket #679).
Don't raise an exception closing an unused named cursor (ticket #716).
Wheel package compiled against PostgreSQL 10.4 libpq and OpenSSL 1.0.2o.

You can install psycopg2 from PyPI or grab the new code from:

Happy hacking!

Psycopg 2.7.4 released

2018-02-08T00:00:00Z

We have released Psycopg version 2.7.4, bringing a few bug fixes... and working out the problem with Wheel packages.

What's the problem with Wheels?

Wheel packages are a Python standard to distribute self-contained binary package. They are especially convenient for packages containing C extensions, such as Psycopg, and for packages depending on external C libraries... such as Psycopg, because the package will contain a binary, pre-compiled, version of the C extension and all the depending libraries (excluding a list of libraries expected to be found on any system and with a versioned ABI, such as libc). Since the release of the wheel packages with psycopg 2.7 it has been possible to install Psycopg without the prerequisites of a C compiler and of Python/Postgres dev packages at build time, and the need of a system libpq at runtime.

Unfortunately, after the packages were released, it was reported of occasional segfaults of Python processes using Psycopg from the Wheel package. This was traced to the use of the Python ssl library concurrently with Psycopg opening connections, for instance in a multithread program opening concurrently https resources and database connections. The problem seems caused by a non-reentrant OpenSSL initialization function (unfortunately invoked by libpq at every connection) and the fact that the Python process ends up binding two different versions of the libssl library: the system one via the Python ssl library (e.g. imported by urllib or requests) and the Wheel one imported by the libpq, in turn imported by psycopg2.

While the problem doesn't affect many users, a library behaving unreliably in combination with part of the stdlib is a situation less than optimal. The workaround is to force installing Psycopg from source, but this must be specified explicitly in the project dependencies (e.g. using the --no-binary flag in the pip command line or in the requirements.txt file); the Python packaging system doesn't really have a way to declare something like "install a package preferably from source"... so we had to create one ourselves.

Starting with Psycopg 2.7.4, we are releasing two different packages on PyPI: psycopg2 and psycopg2-binary. The latter is a Wheels-only package, with a behaviour identical to the classic one – the different name is used only in installation (it is installed by pip install psycopg2-binary, but still imported with import psycopg2 in Python).

For the lifespan of the Psycopg 2.7 cycle, the psycopg2 PyPI package will still contain wheel packages, but starting from Psycopg 2.8 it will become again a source-only package. Starting from Psycopg 2.7.4, if the package is installed as binary from the psycopg2 PyPI entry, a warning will be reported on import:

The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: </docs/install.html#binary-install-from-pypi>.

The choices for the users are then two:

if the program works fine with the Wheel packages, and the convenience of the binary distribution is preferred, it is possible to specify the dependency on the binary package using the psycopg2-binary instead of the psycopg2 PyPI package. No change to the program code is needed;
if there are concerns about the unreliability of the Wheels package, it is advised to force installation from source. This requires the presence of build tools and runtime libraries on the client, but again it requires no change to the code.

We hope this solution will suggest the default use of a reliable version of the library, while still allowing the convenience of a dependencies-free package. Feedback is welcome on the mailing list.

Other changes in this version

Convert fields names into valid Python identifiers in NamedTupleCursor (ticket #211).
Fixed Solaris 10 support (ticket #532).
cursor.mogrify() can be called on closed cursors (ticket #579).
Fixed setting session characteristics in corner cases on autocommit connections (ticket #580).
Fixed MinTimeLoggingCursor on Python 3 (ticket #609).
Fixed parsing of array of points as floats (ticket #613).
Fixed __libpq_version__ building with libpq >= 10.1 (ticket 632).
Fixed rowcount after executemany() with RETURNING statements (ticket 633).
Fixed compatibility problem with pypy3 (ticket #649).
Wheel packages compiled against PostgreSQL 10.1 libpq and OpenSSL 1.0.2n.
Wheel packages for Python 2.6 no more available (support dropped from wheel building infrastructure).

Psycopg 2.7.3.2 released

2017-10-24T00:00:00Z

Following the release of PostgreSQL 10 we have released new binary packages as Psycopg 2.7.3.2. There are no code changes from Psycopg 2.7.3, but the new binary packages ship with the PostgreSQL 10 client library, enabling the use of new features such as multiple hosts in connection string, read-only mode, SCRAM-SHA-256 authentication.

The packages are already available on PyPI and automatically installed by pip install psycopg2 (if your pip is up-to-date). You can grab the source code from:

Happy hacking!

Psycopg 2.7.3.1 released

2017-08-26T00:00:00Z

We have released psycopg2 release 2.7.3.1 as a new build of psycopg 2.7.3. The new build only affects the wheel packages and contains no change in the code.

The release 2.7.3.1 fixes psycopg2-wheels bug #2 which was in turn caused by auditwheel bug #80, resulting in incompatibility with glibc 2.26. The problem only affects Linux wheels users, it doesn't affect Windows, OSX, or user installing psycopg2 from source.

You can install psycopg2 from PyPI or grab the new code from:

Psycopg 2.7.3 released

2017-07-24T00:00:00Z

A quick release to fix a regression found in psycopg 2.7.2:

Restored default timestamptz[] typecasting to Python datetime. Regression introduced in Psycopg 2.7.2 (ticket #578).

You can install psycopg2 from PyPI or grab the new code from:

Happy hacking!

Psycopg 2.7.2 released

2017-07-22T00:00:00Z

Releasing psycopg2 version 2.7.2: a release fixing a host of bugs found in the last 3-4 months.

Summary of changes:

Fixed inconsistent state in externally closed connections (ticket #263, ticket #311, ticket #443). Was fixed in 2.6.2 but not included in 2.7 by mistake.
Fixed Python exceptions propagation in green callback (ticket #410).
Don't display the password in connection.dsn when the connection string is specified as an URI (ticket #528).
Return objects with timezone parsing "infinity" timestamptz (ticket #536).
Dropped dependency on VC9 runtime on Windows binary packages (ticket #541).
Fixed segfault in lobject() when mode=None (ticket #544).
Fixed lobject() keyword argument lobject_factory (ticket #545).
Fixed ReplicationCursor.consume_stream() keepalive_interval argument (ticket #547).
Maybe fixed random import error on Python 3.6 in multiprocess environment (ticket #550).
Fixed random SystemError upon receiving abort signal (ticket #551).
Accept sql.Composable objects in ReplicationCursor.start_replication_expert() (ticket 554).
Parse intervals returned as microseconds from Redshift (ticket #558).
Added Json.prepare() method to consider connection params when adapting (ticket #562).
errorcodes map updated to PostgreSQL 10 beta 1.

You can install psycopg2 from PyPI or grab the new code from:

Happy hacking!

Psycopg 2.7.1 released

2017-03-13T00:00:00Z

A quick bugfix release to solve some teething problems with some of the changes introduced in 2.7:

Ignore None arguments passed to connect() and make_dsn() (ticket #517).
OpenSSL upgraded from major version 0.9.8 to 1.0.2 in the Linux wheel packages (ticket #518).
Fixed build with libpq versions < 9.3 (ticket #520).

Code fresh to grab at:

Enjoy the release!

Psycopg 2.7 released

2017-03-01T00:00:00Z

Finally here! Thank you very much for waiting so long: we have finally released Psycopg 2.7!

Buzzwords:

Faster! Helps generating SQL for repeatedly executed statements and faster Unicode decoding.
Safer! Helps generating dynamic SQL statements with variable table and field names.
Easier! Use the binary package to avoid the need of C compiler, pg_config, libpq required on the clients.
Replication! Support for PostgreSQL physical and logical replication.
Plays-better-with-pgbouncer-at-transaction-pooling-level! This.

You can download as usual from the canonical urls:

Or just use pip:

pip install psycopg2

What's new in psycopg 2.7

New features:

Added psycopg2.sql module to generate SQL dynamically (ticket #308).
Added replication protocol support (ticket #322). Main authors are Oleksandr Shulgin and Craig Ringer, who deserve a huge thank you.
Added parse_dsn() and make_dsn() functions (ticket #321, ticket #363). connect() now can take both dsn and keyword arguments, merging them together.
Added __libpq_version__ and libpq_version() to inspect the version of the libpq library the module was compiled/loaded with (ticket #35, ticket #323).
The attributes notices and notifies can be customized replacing them with any object exposing an append() method (ticket #326).
Adapt network types to ipaddress objects when available. When not enabled, convert arrays of network types to lists by default. The old Inet adapter is deprecated (ticket #317, ticket #343, ticket #387).
Added quote_ident() function (ticket #359).
Added get_dsn_parameters() connection method (ticket #364).
callproc() now accepts a dictionary of parameters (ticket #381).
Give precedence to __conform__() over superclasses to choose an object adapter (ticket #456).
Using Python C API decoding functions and codecs caching for faster unicode encoding/decoding (ticket #473).
executemany() slowness addressed by execute_batch() and execute_values() (ticket #491).
Added async_ as an alias for async to support Python 3.7 where async will become a keyword (ticket #495).
Unless in autocommit, do not use default_transaction_* settings to control the session characteristics as it may create problems with external connection pools such as pgbouncer; use BEGIN options instead (ticket #503).
isolation_level is now writable and entirely separated from autocommit; added readonly, deferrable writable attributes.

Bug fixes:

Fixed error caused by missing decoding LoggingConnection (ticket #483).
Fixed integer overflow in interval seconds (ticket #512).

Other changes:

Dropped support for Python 2.5 and 3.1.
Dropped support for client library older than PostgreSQL 9.1 (but older server versions are still supported).
isolation_level doesn't read from the database but will return ISOLATION_LEVEL_DEFAULT if no value was set on the connection.
Empty arrays no more converted into lists if they don't have a type attached (ticket #506)

Psycopg 2.7 beta 2 released

2017-02-17T00:00:00Z

Hello,

we have released psycopg2 version 2.7 beta 2. This version comes two years after the previous major release so it is packed with new features and improvements; among the main points:

support for replication protocol
helpers to build dynamically SQL statements and manipulate connection strings
speedups in multiple execution and Unicode decoding
better transactions characteristics support

A more complete list of changes is available at /docs/news.html#what-s-new-in-psycopg-2-7

On top of the changes in the code we have addressed the deployment problems found my many inexperienced users, especially on Windows and OSX: psycopg is now distributed as a self-contained wheel package on PyPI, so that:

pip install psycopg2

will not require the presence of C compiler, headers, pg_config, libpq.

Because of all these changes, we ask you kindly to test this psycopg beta release, both its installation and its usage, before we release the stable version, which should happen next week if we don't find any problem. The binary packages of the beta release are on the PyPI testing site: you can install it using:

pip install -U pip # make sure to have an up-to-date pip
pip install -i https://testpypi.python.org/pypi psycopg2==2.7b2

Thank you very much everyone, and I wish to thank especially Oleksandr Shulgin and Craig Ringer for the development of the replication protocol support (the development of which brought along several other features and improvement) and Jason Erickson for the invaluable help with the Windows packages.

Happy testing!

Psycopg 2.6.2 released

2016-07-07T00:00:00Z

Psycopg 2.6.2 has been released. You can get it from:

This is an interim release, packing together one year of bug fixes, before the release 2.7, intended to deliver several new features. Thank you very much to everybody contributing with reports, code, suggestions.

Fixed inconsistent state in externally closed connections (ticket #263, ticket #311, ticket #443).
Report the server response status on errors (such as ticket #281).
Raise NotSupportedError on unhandled server response status (ticket #352).
Allow overriding string adapter encoding with no connection (ticket #331).
The wait_select callback allows interrupting a long-running query in an interactive shell using Ctrl-C (ticket #333).
Fixed PersistentConnectionPool on Python 3 (ticket #348).
Fixed segfault on repr() of an uninitialized connection (ticket #361).
Allow adapting bytes using QuotedString on Python 3 (ticket #365).
Added support for setuptools/wheel (ticket #370).
Fix build on Windows with Python 3.5, VS 2015 (ticket #380).
Fixed errorcodes.lookup initialization thread-safety (ticket #382).
Fixed read() exception propagation in copy_from (ticket #412).
Fixed possible NULL TZ decref (ticket #424).
psycopg2.errorcodes map updated to PostgreSQL 9.5.

Psycopg 2.6.1 released

2015-06-16T00:00:00Z

Psycopg 2.6.1 has been released. You can get it from:

The most important bug fixed in this release is the libcrypto callbacks conflict between libpq and Python (ticket #290), fixed by Jan Urbański: thank you very much. Other bugs fixed:

Lists consisting of only None are escaped correctly (ticket #285).
Correctly unlock the connection after error in flush (ticket #294).
Fixed MinTimeLoggingCursor.callproc() (ticket #309).

Psycopg 2.6 and 2.5.5 released

2015-02-09T00:00:00Z

We have just released two Psycopg versions: 2.5.5 containing a few bug fixes and 2.6 introducing some new features.

The main new feature in Psycopg 2.6 is the support for the large objects 64 bits API, allowing access to large objects of more than 2GB size. Note that the feature is only supported on 64 bits clients and with at least PostgreSQL 9.3 server and client library.

I wish to thank all the people who have contributed to these releases, and especially Blake Rouse and the MAAS Team for the development of the large objects 64 bits API support.

Psycopg 2.6 source package, signature
Psycopg 2.5.5 source package, signature

This is the full list of changes for the new versions:

Psycopg 2.6 new features

Added support for large objects larger than 2GB.
Python time objects with a tzinfo specified and PostgreSQL timetz data are converted into each other (ticket #272).

Psycopg 2.6 bug fixes

Json apapter's str() returns the adapted content instead of the repr() (ticket #191).

Psycopg 2.5.5 bug fixes

Named cursors used as context manager don't swallow the exception on exit (ticket #262).
cursor.description can be pickled (ticket #265).
Propagate read error messages in COPY FROM (ticket #270).
PostgreSQL time 24:00 is converted to Python 00:00 (ticket #278).

A further note for the packagers: the Psycopg 2.6 source tarball doesn't contain the HTML rendered documentation anymore: you will find Makefile targets to easily build it.

Psycopg 2.5.4 released

2014-08-30T00:00:00Z

Psycopg 2.5.4 has been released. You can get it from:

This version supports the new jsonb PostgreSQL type out-of-the-box. And of course there are a few bug fixed:

Added jsonb support for PostgreSQL 9.4 (ticket #226).
Fixed segfault if COPY statements are passed to execute() instead of using the proper methods (ticket #219).
Force conversion of pool arguments to integer to avoid potentially unbounded pools (ticket #220).
Cursors WITH HOLD don't begin a new transaction upon move/fetch/close (ticket #228).
Cursors WITH HOLD can be used in autocommit (ticket #229).
callproc() doesn't silently ignore an argument without a length.
Fixed memory leak with large objects (ticket #256).
The internal _psycopg.so module can be imported stand-alone (to allow modules juggling such as the one described in ticket #201).

Have a good weekend!

Cancelling PostgreSQL statements from Python

2014-07-20T00:00:00Z

Cancelling a long running query from Python is not something that happens automatically: the libpq doesn't react to Python signals so the only way to stop a query is to run a pg_cancel_backend from another process. Killing the Python process won't cancel the query: it will run until completion and then rolled back. This makes working with long-running query from the Python interpreter somewhat frustrating.

Using psycopg in green mode moves the waiting from the libpq C code to Python: this gives Python some chance of interaction: it is possible for instance to catch a ctrl-c and send a cancel request:

from select import select
from psycopg2.extensions import POLL_OK, POLL_READ, POLL_WRITE

def wait_select_inter(conn):
    while 1:
        try:
            state = conn.poll()
            if state == POLL_OK:
                break
            elif state == POLL_READ:
                select([conn.fileno()], [], [])
            elif state == POLL_WRITE:
                select([], [conn.fileno()], [])
            else:
                raise conn.OperationalError(
                    "bad state from poll: %s" % state)
        except KeyboardInterrupt:
            conn.cancel()
            # the loop will be broken by a server error
            continue

psycopg2.extensions.set_wait_callback(wait_select_inter)

An interactive session would look like:

>>> cnn = psycopg2.connect('')
>>> cur = cnn.cursor()
>>> cur.execute("select pg_sleep(10)")
^C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
QueryCanceledError: canceling statement due to user request

The connection is now in error state, but a cnn.rollback() would make it working again.

Psycopg 2.5.3 Released

2014-05-13T00:00:00Z

Psycopg 2.5.3 has been released. You can get it from:

This version contains several bug fixes over the previous release 2.5.2:

Work around pip issue #1630 making installation via pip -e git+url impossible (ticket #18).
Copy operations correctly set the cursor.rowcount attribute (ticket #180).
It is now possible to call get_transaction_status() on closed connections.
Fixed unsafe access to object names causing assertion failures in Python 3 debug builds (ticket #188).
Mark the connection closed if found broken on poll() (from ticket #192 discussion)
Fixed handling of dsn and closed attributes in connection subclasses failing to connect (from ticket #192 discussion).
Added arbitrary but stable order to Range objects, thanks to Chris Withers (ticket #193).
Avoid blocking async connections on connect (ticket #194). Thanks to Adam Petrovich for the bug report and diagnosis.
Don't segfault using poorly defined cursor subclasses which forgot to call the superclass init (ticket #195).
Mark the connection closed when a Socket connection is broken, as it happens for TCP connections instead (ticket #196).
Fixed overflow opening a lobject with an oid not fitting in a signed int (ticket #203).
Fixed handling of explicit default cursor_factory=None in connection.cursor() (ticket #210).
Fixed possible segfault in named cursors creation.
Fixed debug build on Windows, thanks to James Emerton.

Thank you very much to everybody helping with this release!

Psycopg 2.5.2 released

2014-01-07T00:00:00Z

Psycopg 2.5.2 has been released. You can get it from:

This version contains a few bug fixes over the previous release 2.5.1:

Fixed segfault pickling the exception raised on connection error (ticket #170).
Meaningful connection errors report a meaningful message, thanks to Alexey Borzenkov (ticket #173).
Manually creating lobject with the wrong parameter doesn't segfault (ticket #187).

Thank you very much to all the people who helped during the development!

Psycopg 2.5.1 released

2013-06-23T00:00:00Z

Psycopg 2.5.1 has been released. You can get it from:

The version contains a few bug fixes over the previous 2.5:

Fixed build on Solaris 10 and 11 where the round() function is already declared (ticket #146).
Fixed comparison of Range with non-range objects (ticket #164). Thanks to Chris Withers for the patch.
Fixed double-free on connection dealloc (ticket #166). Thanks to Gangadharan S.A. for the report and fix suggestion.

Happy hacking!

Psycopg 2.5 released

2013-04-07T00:00:00Z

We are happy to introduce the release 2.5 of Psycopg, packed with several juicy new features!

Here are a few highlights of the release:

JSON adapter

The JSON adapter allows easy exchange of unstructured data with the database:

>>> cur.execute(
...     """select '{"a":[1,2,3],"b":[4,5,6]}'::json""")
>>> cur.fetchone()[0]
{u'a': [1, 2, 3], u'b': [4, 5, 6]}

It works with PostgreSQL 9.2 native JSON data type, with previous versions' extension modules and, in a pinch, with regular text too. You can feed JSON data to PostgreSQL using a specific adapter:

curs.execute("insert into mytable (jsondata) values (%s)",
    [Json({'a': 100})])

or you can be more aggressive and ask for automatic JSON serialization for every type you want to support, dictionaries for instance:

psycopg2.extensions.register_adapter(dict, Json)
curs.execute("insert into mytable (jsondata) values (%s)",
    [{'a': 100}])

Range adapters

Other handy adapters for PostgreSQL 9.2 are the Range data type adapters family: ranges are returned as an object allowing access to their properties:

>>> cur.execute("""select '[10,20)'::int4range""")
>>> r = cur.fetchone()[0]
>>> r.lower
10
>>> r.upper_inc
False

Of course the Python object can be used as arguments in query parameters:

r = DateRange(date(2013,1,1), date(2013,2,1), '[)')
cur.execute("select * from events where %s @> date", [r])

Built-in range types are supported out-of-the-box, while the function register_range() allows handling user-defined range types.

Connections and cursors as context managers

A recent DBAPI extension has standardized the use of connections and cursors as context managers: it is now possible to use an idiom such as:

with psycopg2.connect(DSN) as conn:
    with conn.cursor() as curs:
       curs.execute(SQL)

with the intuitive behaviour: when the cursor block exits the cursor is closed; when the connection block exits normally the current transaction is committed, if it exits with an exception instead the transaction is rolled back, in either case the connection is ready to be used again (FIXED: the connection is NOT closed as originally stated).

Other new features in Psycopg 2.5

Added Diagnostics object to get extended info from a database error. Many thanks to Matthew Woodcraft for the implementation (ticket #149).
Added connection.cursor_factory attribute to customize the default object returned by cursor().
Added support for backward scrollable cursors. Thanks to Jon Nelson for the initial patch (ticket #108).
Added a simple way to customize casting of composite types into Python objects other than namedtuples. Many thanks to Ronan Dunklau and Tobias Oberstein for the feature development.
connection.reset() implemented using DISCARD ALL on server versions supporting it.

Bug fixes:

Properly cleanup memory of broken connections (ticket #148).
Fixed bad interaction of setup.py with other dependencies in Distribute projects on Python 3 (ticket #153).

Other changes:

Added support for Python 3.3.
Dropped support for Python 2.4. Please use Psycopg 2.4.x if you need it.
errorcodes map updated to PostgreSQL 9.2.
Dropped Zope adapter from source repository. ZPsycopgDA has now its own project at <https://github.com/psycopg/ZPsycopgDA>.

Psycopg 2.4.6 released

2012-12-12T00:00:00Z

I'm happy to announce the release of Psycopg 2.4.6: a huge thank you to the many contributors.

This is a bugfix release, introducing no new feature. There are several small corrections in different areas (copy, adaptation, use of extra cursors, stability). The biggest improvements are with the Zope adapter: Zope users using previous 2.4.x versions are encouraged to update to version 2.4.6 soon.

What's new in psycopg 2.4.6

Fixed 'cursor()' arguments propagation in connection subclasses and overriding of the 'cursor_factory' argument. Thanks to Corry Haines for the report and the initial patch (ticket #105).
Dropped GIL release during string adaptation around a function call invoking a Python API function, which could cause interpreter crash. Thanks to Manu Cupcic for the report (ticket #110).
Close a green connection if there is an error in the callback. Maybe a harsh solution but it leaves the program responsive (ticket #113).
'register_hstore()', 'register_composite()', 'tpc_recover()' work with RealDictConnection and Cursor (ticket #114).
Fixed broken pool for Zope and connections re-init across ZSQL methods in the same request (tickets #123, #125, #142).
connect() raises an exception instead of swallowing keyword arguments when a connection string is specified as well (ticket #131).
Discard any result produced by 'executemany()' (ticket #133).
Fixed pickling of FixedOffsetTimezone objects (ticket #135).
Release the GIL around PQgetResult calls after COPY (ticket #140).
Fixed empty strings handling in composite caster (ticket #141).
Fixed pickling of DictRow and RealDictRow objects.

Prepared statements in Psycopg

2012-10-01T00:00:00Z

Although the libpq library supports prepared statements, psycopg2 doesn't offer yet a direct way to access the relevant functions. This will probably change in the future, but in the meantime it is possible to use prepared statements in PostgreSQL using the PREPARE SQL command.

Whenever you have a loop where the same parametrized query or command is executed:

cur = conn.cursor()
for i, j in parameters:
    cur.execute(
        "select * from tables where i = %s and j = %s",
        (i, j))
    for record in cur:
        do_something_with(record)

you can actually ask PostgreSQL to prepare the plan in advance and use it, saving time in the inner loop:

cur = conn.cursor()
cur.execute(
    "prepare myplan as "
    "select * from tables where i = $1 and j = $2")
for i, j in parameters:
    cur.execute("execute myplan (%s, %s)", (i, j))
    for record in cur:
        do_something_with(record)

The time saved could be relevant for complex queries which are fast to execute; for queries that instead take several seconds to complete, the planning time is probably negligible so there wouldn't be much to save. Note that the query passed to PREPARE uses Postgres-style placeholders ($1, $2...) instead of the familiar Python-style %s or %(name)s.

So is psycopg useless in this case? Is the burden of executing PREPARE on the poor user? Well, it's actually easy to write a cursor subclass implementing prepared statements. The proposed PreparingCursor doesn't PREPARE each statement passed to execute(): this would be harmful as it involves an extra roundtrip to the server and the plan for a prepared statements is sometimes less efficient than one calculated with the knowledge of the real parameters used. So the proposed class exposes an explicit prepare() method: it takes a query (written with Python-style placeholders, so exactly the one you would have used with execute()), replaces it with Postgres-style placeholders and PREPAREs it. In further calls to execute() the query can be omitted: in this case (or if the query is the one prepared) the prepared statement is executed instead.

Using the PreparingCursor the above example could be written as:

cur = conn.cursor(cursor_factory=PreparingCursor)
cur.prepare(
    "select * from tables where i = %s and j = %s")
for i, j in parameters:
    cur.execute((i, j))
    for record in cur:
        do_something_with(record)

The preparing cursor also overrides executemany(): in this case the query is always prepared as it is assumed that it will be executed more than once. Other extensions are a prepared attribute, returning the prepared statement if any, and a deallocate() method to release the prepared statement (which would be deleted anyway at the end of the session).

The preparing cursor may find its way into a future psycopg2 release, but its design is not finalized yet and several details, both in the interface and the implementation, could be done in different ways. So please use it and give us feedback: we'll use it to design the optimal implementation for Psycopg!

Psycopg 2.4.5 released

2012-03-29T00:00:00Z

Many thanks to everybody that contributed with bug reports and comments to this release!

What's new in psycopg 2.4.5

The close() methods on connections and cursors don't raise exceptions if called on already closed objects.
Fixed fetchmany() with no argument in cursor subclasses (ticket #84).
Use lo_creat() instead of lo_create() when possible for better interaction with pgpool-II (ticket #88).
Error and its subclasses are picklable, useful for multiprocessing interaction (ticket #90).
Better efficiency and formatting of timezone offset objects thanks to Menno Smits (tickets #94, #95).
Fixed rownumber during iteration on cursor subclasses. Regression introduced in 2.4.4 (ticket #100).
Added support for inet arrays.
Fixed commit() concurrency problem (ticket #103).
Codebase cleaned up using the GCC Python plugin's static analysis tool, which has revealed several unchecked return values, possible NULL dereferences, reference counting problems. Many thanks to David Malcolm for the useful tool and the assistance provided using it.

Psycopg 2.4.4 released

2011-12-19T00:00:00Z

After a short discussion on this list we decided to change the definitions of isolation levels to make sure old code using numeric constants (both psycopg1 and psycopg2) continue to works. Other small fixes are included in the release: see below for details.

What's new in psycopg 2.4.4

register_composite() also works with the types implicitly defined after a table row, not only with the ones created by CREATE TYPE.
Values for the isolation level symbolic constants restored to what they were before release 2.4.2 to avoid breaking apps using the values instead of the constants.
Named DictCursor/RealDictCursor honour itersize (ticket #80).
Fixed rollback on error on Zope (ticket #73).
Raise DatabaseError instead of Error with empty libpq errors, consistently with other disconnection-related errors: regression introduced in release 2.4.1 (ticket #82).

Psycopg 2.4.3 released

2011-12-12T00:00:00Z

Mostly a bugfix release, with as usual a couple of small feature added:

Here is what's new in this release:

connect() supports all the keyword arguments supported by the database
Added new_array_type() function for easy creation of array typecasters.
Added support for arrays of hstores and composite types (ticket #66).
Fixed segfault in case of transaction started with connection lost (and possibly other events).
Fixed adaptation of Decimal type in sub-interpreters, such as in certain mod_wsgi configurations (ticket #52).
Rollback connections in transaction or in error before putting them back into a pool. Also discard broken connections (ticket #62).
Lazy import of the slow uuid module, thanks to Marko Kreen.
Fixed NamedTupleCursor.executemany() (ticket #65).
Fixed --static-libpq setup option (ticket #64).
Fixed interaction between RealDictCursor and named cursors (ticket #67).
Dropped limit on the columns length in COPY operations (ticket #68).
Fixed reference leak with arguments referenced more than once in queries (ticket #81).
Fixed typecasting of arrays containing consecutive backslashes.
errorcodes map updated to PostgreSQL 9.1.

Psycopg 2.4.2 released

2011-06-12T00:00:00Z

Psycopg 2.4.2 has been released: it brings a few small but interesting new features, and a lot of bug fixes.

Transaction control has been overhauled: a new connection method set_session() allows setting all the session properties affecting the transactions behaviour: the isolation level but it can also be used to have auto-commit, read-only, and deferrable transactions. You can use for example:

conn.set_session('read committed')
conn.set_session(readonly=True, autocommit=True)
conn.set_session('serializable',
                 readonly=True, deferrable=True)

There is also a welcome improvement for all the users thinking that

conn.set_isolation_level(
    psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)

was an excessively verbose syntax: a new read/write property allows to set

conn.autocommit = True

a much handier syntax for an often used connection property.

The improvements to the transactions control are not only at interface level: Psycopg doesn't require any more setup queries when connecting to a database. A sequence of statements:

import psycopg2
conn = psycopg2.connect('')
curs = conn.cursor()
curs.execute('SELECT 1')
curs.execute('SELECT 2')
conn.commit()

in older versions of the library would have resulted in the following commands sent to the backend:

SHOW default_transaction_isolation
SET DATESTYLE TO 'ISO'
SHOW client_encoding
BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
SELECT 1
SELECT 2
COMMIT

In Psycopg 2.4.2 the only commands sent are instead the essential:

BEGIN
SELECT 1
SELECT 2
COMMIT

with the BEGIN/COMMIT obviously omitted in autocommit mode (the datestyle and encoding statements were already dropped in 2.3).

The new release also brings a lot of bug fixes, so we encourage updating soon, particularly if you use Psycopg in multithread programs:

Fixed bug with multithread code potentially causing loss of sync with the server communication or lock of the client (ticket #55).
Don't fail import if mx.DateTime module can't be found, even if its support was built (ticket #53): a fix for the "ghost mx.DateTime" issue reported in virtualenv.
Fixed escape for negative numbers prefixed by minus operator (ticket #57).
Fixed GC race condition during copy in multithread programs, potentially resulting in refcount errors. Fixed by Dave Malcolm (ticket #58, Red Hat Bug 711095).
Trying to execute concurrent operations on the same connection through concurrent green thread results in an error instead of a deadlock.
Fixed detection of pg_config on Windows. Report and fix, plus some long needed setup.py cleanup by Steve Lacy: thanks!

Building Psycopg on Windows using MinGW

2011-06-05T00:00:00Z

My goal was to install Psycopg on Windows using MinGW and the PostgreSQL binary package.

I have used the MinGW GCC binaries packaged by Giovanni Bajo. The package takes care of a lot of details, for instance registering MinGW as default compiler for Python, plus some magic I don't even want to know, and makes the entire process simple enough.

The first step is to ensure that setup.py can find pg_config. There is a bug causing it not to be found if it is in the path: it will be fixed in Psycopg 2.4.2. On a few older versions you will have to specify the pg_config path in the setup.cfg or by the --pg-config option, e.g.

python setup.py build_ext --pg-config=C:\path\to\pg_config.exe build

The library built depends on libpq.dll, so at runtime this library should be available, e.g. on the path or it may be copied in the psycopg2 directory. The libpq in turn depends on a few other dlls, all found in the PostgreSQL bin directory: libeay32.dll, ssleay32.dll and libintl-8.dll: they should be made available to the client as well. Unfortunately if any of these libraries is missing you will only get an "ImportError: dll load failed". The problem is very easy to debug using a dependency walker.

Another problem you may find is building for Python 2.6 and newer: some MinGW versions ship with a broken msvcr90 lib, and the result is again a rather unhelpful ImportError. the dependency walker is useful in this case too, showing the missing localtime function in the library. The bug was reported in the issue 3308: my solution was to download a newer MinGW and use the libmsvcr90.a from there.

So, overall, it can be done, but the result still depends on many dlls. I'm torn if the solution could be to have all the dependencies copied in the package directory. Of course you can still use Jason's binary package: he builds both the libpq and the openssl as static libs and creates a self-contained psycopg, which is probably the handiest result, but can't be obtained only using the PostgreSQL binaries.

Psycopg 2.4.1 released

2011-05-11T00:00:00Z

Hi *,

Daniele stacked another round of fixes on the devel branch, so it is time for another release. So, as always, kudos to Daniele and here are the direct download links:

Release notes attached, as always. And al always, have fun,

federico

What's new in psycopg 2.4.1

Use own parser for bytea output, not requiring anymore the libpq 9.0 to parse the hex format.
Don't fail connection if the client encoding is a non-normalized variant. Issue reported by Peter Eisentraut.
Correctly detect an empty query sent to the backend (ticket #46).
Fixed a SystemError clobbering libpq errors raised without SQLSTATE. Bug vivisectioned by Eric Snow.
Fixed interaction between NamedTuple and server-side cursors.
Allow to specify --static-libpq on setup.py command line instead of just in 'setup.cfg'. Patch provided by Matthew Ryan (ticket #48).

Psycopg 2.4 released

2011-02-27T00:00:00Z

Hi *,

this is probably one of the best psycopg releases ever. Daniele, Jason and all the others that sent patches did an impressive work to have psycopg build and work flawlessly on all the supported platforms (well.. we can probably do a little bit better on MacOS but everything else is almost perfect). So here it is (followed by NEWS excerpt, as always):

Have fun,

federico

What's new in psycopg 2.4

Added support for Python 3.1 and 3.2. The conversion has also brought several improvements:

Added 'b' and 't' mode to large objects: write can deal with both bytes strings and unicode; read can return either bytes strings or decoded unicode.
COPY sends Unicode data to files implementing 'io.TextIOBase'.
Improved PostgreSQL-Python encodings mapping.
Added a few missing encodings: EUC_CN, EUC_JIS_2004, ISO885910, ISO885916, LATIN10, SHIFT_JIS_2004.
Dropped repeated dictionary lookups with unicode query/parameters.

Improvements to the named cusors:

More efficient iteration on named cursors, fetching 'itersize' records at time from the backend.
The named cursors name can be an invalid identifier.

Improvements in data handling:

Added 'register_composite()' function to cast PostgreSQL composite types into Python tuples/namedtuples.
Adapt types 'bytearray' (from Python 2.6), 'memoryview' (from Python 2.7) and other objects implementing the "Revised Buffer Protocol" to 'bytea' data type.
The 'hstore' adapter can work even when the data type is not installed in the 'public' namespace.
Raise a clean exception instead of returning bad data when receiving bytea in 'hex' format and the client libpq can't parse them.
Empty lists correctly roundtrip Python -> PostgreSQL -> Python.

Other changes:

'cursor.description' is provided as named tuples if available.
The build script refuses to guess values if 'pg_config' is not found.
Connections and cursors are weakly referenceable.

Bug fixes:

Fixed adaptation of None in composite types (ticket #26). Bug report by Karsten Hilbert.
Fixed several reference leaks in less common code paths.
Fixed segfault when a large object is closed and its connection no more available.
Added missing icon to ZPsycopgDA package, not available in Zope 2.12.9 (ticket #30). Bug report and patch by Pumukel.
Fixed conversion of negative infinity (ticket #40). Bug report and patch by Marti Raudsepp.

Psycopg 2.4 beta1 released

2011-02-06T00:00:00Z

Hi *,

me and Daniele, we just went through his series of patches (a loooot of patches) and we're ready to release a beta version of the new psycopg. Apart the usual changes and enhancements (detailed below) this is the first version that supports Python 3 (thanks to, guess who? :)

The NEWS excerpt:

New features and changes

Added register_composite() function to cast PostgreSQL composite types into Python tuples/namedtuples.
More efficient iteration on named cursors.
The build script refuses to guess values if pg_config is not found.
Connections and cursors are weakly referenceable.
Added 'b' and 't' mode to large objects: write can deal with both bytes strings and unicode; read can return either bytes strings or decoded unicode.
COPY sends Unicode data to files implementing io.TextIOBase.
The build script refuses to guess values if pg_config is not found.
Improved PostgreSQL-Python encodings mapping. Added a few missing encodings: EUC_CN, EUC_JIS_2004, ISO885910, ISO885916, LATIN10, SHIFT_JIS_2004.
Dropped repeated dictionary lookups with unicode query/parameters.
Empty lists correctly roundtrip Python -> PostgreSQL -> Python.

Bug fixes

Fixed adaptation of None in composite types (ticket #26). Bug report by Karsten Hilbert.
Fixed several reference leaks in less common code paths.
Fixed segfault when a large object is closed and its connection no more available.
Added missing icon to ZPsycopgDA package, not available in Zope 2.12.9 (ticket #30). Bug report and patch by Pumukel.

Have fun,

federico

psycopg2 porting to Python 3: a report

2011-01-24T00:00:00Z

I've mostly finished the porting of psycopg2 to Python 3. Here is a report of what done and what can be improved.

The code is available in the python3 branch of the repository available on <https://github.com/dvarrazzo/psycopg>. The code is compatible with both python2 and python3: the Python code is in py2 syntax: setup.py processes it with 2to3 before installation. The C code uses a macros portability layer (in psycopg/python.h) to have py2 and py3 code unified.

A big chunk of the porting is by Martin von Löwis (who I thank wholeheartedly): who provided a big patch back in 2008 (against 2.0.9, IIRC). Unfortunately psycopg code diverged without the patch being merged or maintained, so I basically used his macros but re-did the work from scratch, refactoring the code instead of patching many repetitive parts. On the pro side, since then, psycopg gained many more tests.

Large part of the porting has been mechanical, nothing to say about that. What has required decisions has been instead the string processing: Py3 uses extensively Unicode, but the communication with the libpq is performed in char*; being psycopg a programmable interface the point in which the conversion happens changes how adapters and typecasters should be written.

Adapters

they are objects wrapping any Python object and returning a SQL representation to be passed to the libpq. The adapters may have returned either a str or an unicode, but a critical step is to pass through libpq functions to have string and binary data escaped (e.g. PQescapeStringConn). Because these functions are defined char* -> char*, what makes sense for me was to force adapters to return bytes: having them returning unicode would mean that unicode strings should have been:

converted to bytes
escaped by the libpq
converted back to unicode to be returned from the adapter (but at this point which encoding to use is not clear)
merged to the query
converted to bytes again to be sent to the socket

The double encoding seems unnecessary, so I prefer to have adapters to return bytes. Having them free to return either bytes or unicode makes writing composite adapters trickier and more error prone, so my decision is to raise an exception if after adaptation a non-bytes object is returned.

Having adapted objects as bytes means that the arguments must be merged to the query as bytes: this operation is performed by not much more than a query % args. Unfortunately the % operator is not available for bytes, so I have ported the PyString_Format from Python 2.7 and adapted to work with the bytes (the Python license seems allowing mixing derived code with the LGPL without problems).

Typecasters

These are function performing the opposite: they take the PostgreSQL representation of a value and convert it into a Python object. They receive bytes from the libpq of course. What I have currently implemented is to convert this string to unicode before passing it to the Python functions: because in Python the parsing functions mostly take strings as argument (meaning unicode in py3), passing bytes to the typecasters would have meant that each of them should have implemented about the same boilerplate, something like:

def caster(value, curs):
    value = value.decode(
        psycopg.encodings[curs.connection.encoding])

but only in Py3, not in Py2. The current approach (passing an already decoded object) makes writing the typecasters easier, but has the drawback of making impossible to write a Python typecaster for a binary type (but I don't think there is really the need for such caster) and it is kinda inconsistent with the adapters (dealing with bytes). What option would be better?

COPY

Copy operations deal with Python files or file-like objects. In input (COPY IN) both unicode and bytes files are accepted; unicode is converted in the connection encoding. In output (COPY OUT)... oops: reviewing now I see I've overlooked this part: as it is now the data (bytes) from the libpq are passed to file.write() using PyObject_CallFunction(func, "s#", buffer, len). But this implies that buffer is decoded from utf8 in Py3, so it would break if the connection encoding was different. I've done a quick check and in Py3 a file open in text mode doesn't accept bytes, while one open in binary mode doesn't accept unicode. Uhm... what could we pass this file? Is there an interface in Python 3 to know if a file is binary or text? Added ticket #36.

Large Objects

These are open using a mode string such as "r", "w", "rw". I have added a format letter pretty much as the open() function in Py3: it can be "b" or "t". In binary mode the file always returns bytes (str in py2, bytes [edit: not unicode] in py3). In text mode it always returns unicode (unicode in py2, str in py3). The default is "b" in py2, "t" in py3. writing to the file accepts both str and unicode. This means that in Py2 everything is compatible, but there are a few features added (unicode communication) and it's easy to write portable code by specifying the mode "b" or "t".

Other random details

in py2 psycopg uses basic string as default, and unicode must be chosen specifically (e.g. registering the adapter, passing a unicode=True to certain functions etc.) In py3 there is no such choice and unicode is returned where there used to be a choice.
bytea fields are returned as MemoryView, from which is easy to get bytes.
"secondary strings" (notices, notifications, errors...) are decoded in the connection encoding, but I'm not be 100% sure that this will be always right, so the decoding is forgiving: decode(x, 'replace') for them.

This should be pretty much everything about the Py3 porting. Comments are welcome, above all on the open points (typecasters and COPY OUT), but if there is anything to point out I'd be happy to know. Best option would be to discuss on the mailing list. See you there!

Update: Psycopg 2.4 beta 1 has been released with Python 3 support.

New Psycopg Mailing List Online!

2011-01-08T00:00:00Z

After a long while Psycopg has a Mailing List again!

You are welcome to subscribe, either if you are an user dealing with the first stumbling blocks (albeit a look at the documentation or the FAQ wouldn't hurt!) or if you want to contribute to the psycopg2 development, about which there are several upcoming news.

To post, send mail to <psycopg@postgresql.org>. Yes, we are proud to be hosted on the mighty shoulders of the PostgreSQL Infrastructure team. I want to thank them and the PostgreSQL Global Development Group wholeheartedly for making this happen.

See you there!

Psycopg 2.3.2 released

2010-12-20T00:00:00Z

Hello,

just released Psycopg 2.3.2. The release fixes a bug reported in 2.3.0 and 2.3.1 preventing Psycopg to connect to pgBouncer. Thanks to Marti Raudsepp for the bug report and the patch.

Download from here:

Cheers!

Psycopg 2.3.1 released

2010-12-04T00:00:00Z

Hello,

just released Psycopg 2.3.1. No new feature since release 2.3.0: the only fix is for a build issue on CentOS 5.5 x86_64 (ticket #23).

Download URLs:

Cheers!

Articles

Psycopg 3.3 released

Template string queries

More flexible composite adaptation

Solving the fetchone() annoyance with type checkers

Improvements to the connection pools

...And more!

Your help is welcome

Automatic async to sync code conversion

This is so boring that...

Abstract Syntax Tree

Du AST Mich

Problems with sleep()

When everything else fail

Conversion methodology

The final result

Psycopg 3.2 released

Numpy scalars support

PostgreSQL parameters

Scalar row factory

Libpq 17 features

Easier interaction with notifications

Less work for us!

We need your help!

Pipeline mode in Psycopg

Using the pipeline mode in Psycopg

What is the pipeline mode for?

Interlude: tracing

To pipeline or not to pipeline

How does it work?

Socket communication

Queueing work, processing results

Integration with high-level features: transactions

Nested pipelines

Pipelined transactions

Psycopg 3.1 released

Pipeline mode

Client-binding cursors

Enum adaptation

In partnership with CockroachDB

executemany() improvements

And many more improvements

Thank you very much!

Psycopg 3.0 released

Psycopg 3.0 beta 1 released!

Building a Django driver for Psycopg 3

Server-side parameters binding

Server-side binding doesn't work for all the queries

GROUP BY/ORDER BY with parameters

A different package organisation

Takeaway points

Psycopg 2.9 released

Designing a connection pool for psycopg3

Client interface

Context manager

Blocking behaviour

Connection configuration

Exclusive connections?

Internal behaviour

Pool worker

What to do in case of connection failure?

Connections usage pattern

Proposed API

Key configuration parameters

Thoughts?

The psycopg3 adaptation system

Server-side binding

A new adaptation system

Customising psycopg3 adaptation

Project sponsorship

The new COPY support in psycopg3

Binary format

Block-level COPY

Asynchronous COPY

Project sponsorship

Psycopg 2.8.6 released

Psycopg 2.8.5 released

Psycopg 2.8.4 released

Psycopg 2.8.3 released

Psycopg 2.8.1, 2.8.2 released

Solving the `fetchone()` annoyance with type checkers

Problems with `sleep()`

`executemany()` improvements