This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------ce3546e9f767c4f407662caac2a8083d8c4baf7a1f1fc141736db6f000bc5b55 Content-Type: multipart/mixed;boundary=---------------------c50b1a0d029d02ab0c0c864f7ca0375d

-----------------------c50b1a0d029d02ab0c0c864f7ca0375d Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset=utf-8

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu=
Linux.

/*=C2=A0* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
=C2=A0*
=C2=A0* g++ python_thread_test.cpp `python3-config --cflags --libs --embed=
`
=C2=A0* ./a.out
=C2=A0*
=C2=A0* uname:
=C2=A0* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linu=
x
=C2=A0*/

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void perform_long_operation() {
=C2=A0 =C2=A0 // Simulate a long-running task
=C2=A0 =C2=A0 std::this_thread::sleep_for(std::chrono::seconds(5));
}

void thread_function() {
=C2=A0 =C2=A0 // Ensure this thread has the GIL
=C2=A0 =C2=A0 PyGILState_STATE gstate =3D PyGILState_Ensure();

=C2=A0 =C2=A0 // Execute some Python code
=C2=A0 =C2=A0 PyRun_SimpleString("print('Hello from std::thread!')");

=C2=A0 =C2=A0 // Release the GIL for long operation
=C2=A0 =C2=A0 Py_BEGIN_ALLOW_THREADS
=C2=A0 =C2=A0 perform_long_operation();
=C2=A0 =C2=A0 Py_END_ALLOW_THREADS

=C2=A0 =C2=A0 // Re-acquire the GIL and execute more Python code
=C2=A0 =C2=A0 gstate =3D PyGILState_Ensure();
=C2=A0 =C2=A0 PyRun_SimpleString("print('Thread operation completed!')");

=C2=A0 =C2=A0 // Release the GIL
=C2=A0 =C2=A0 PyGILState_Release(gstate);
}

int main() {
=C2=A0 =C2=A0 // Initialize the Python Interpreter
=C2=A0 =C2=A0 Py_Initialize();

=C2=A0 =C2=A0 // Create a vector of threads
=C2=A0 =C2=A0 std::vector<std::thread> threads;

=C2=A0 =C2=A0 // Launch threads
=C2=A0 =C2=A0 for (int i =3D 0; i < 5; ++i) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 threads.push_back(std::thread(thread_function)=
);
=C2=A0 =C2=A0 }

=C2=A0 =C2=A0 // Join threads
=C2=A0 =C2=A0 for (auto& t : threads) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 t.join();
=C2=A0 =C2=A0 }

=C2=A0 =C2=A0 // Finalize the Python Interpreter
=C2=A0 =C2=A0 Py_Finalize();

=C2=A0 =C2=A0 return 0;
}



Tomas Ukkonen
-----------------------c50b1a0d029d02ab0c0c864f7ca0375d--

--------ce3546e9f767c4f407662caac2a8083d8c4baf7a1f1fc141736db6f000bc5b55 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wsBzBAEBCAAnBYJmiutFCZCaEYwMt4b97BYhBJbmzyegv3Wo8O5gpZoRjAy3 hv3sAACH0wgAk6xX7goQyDG6b2zgFvqJ0HoSqH5qjNyXyKb8JrdWMS08dw53 aOwYhRvRRvNK17zv1RPp8G8ROjg9S3zjuURb4ChofqUZk8iHKDFG782JTdPG EE1XP8e/j0VNHwRJZyif/cKcu6JUnVlISv6xMKCvBxCC3dDmVEi2S3YQEC3T VJCw2XKpP1TEVXoDpnXgBIbHaGlm+EGCADIuZqXRI3idHnsPM+6tQVRX8R8y L3/7jooYtXx7UIoiX6t3o5TWc42lD/eCmWyDNHAxZHz+8KiolxCXi+IDy9RD Kpwwvj3gPUaQlk7HjTb7EkuoU9xl4V63QwfruvCtDX3gJak4+ac4sg==
=0jND
-----END PGP SIGNATURE-----


--------ce3546e9f767c4f407662caac2a8083d8c4baf7a1f1fc141736db6f000bc5b55--


--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------cea9a307b89f84c86fe7aee189ffbbfe37df1fbb2efb3b4f35743011c9dd2ff9 Content-Type: multipart/mixed;boundary=---------------------9e29034e00ea96b7d10a477458ea4ed4

-----------------------9e29034e00ea96b7d10a477458ea4ed4 Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset=utf-8

Hi

There was a bug in the example code. I fixed it and it STILL deadlocks (my=
larger software project deadlocks when I call python from C++).

Updated code:

/*=C2=A0* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
=C2=A0*
=C2=A0* g++ python_thread_test.cpp `python3-config --cflags --libs --embed=
`
=C2=A0* ./a.out
=C2=A0*
=C2=A0* uname:
=C2=A0* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linu=
x
=C2=A0*/

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>


void thread_function() {
=C2=A0 =C2=A0 // Ensure this thread has the GIL
=C2=A0 =C2=A0 PyGILState_STATE gstate =3D PyGILState_Ensure();

=C2=A0 =C2=A0 // Execute some Python code
=C2=A0 =C2=A0 PyRun_SimpleString("print('Hello from std::thread!')");

=C2=A0 =C2=A0 // Release the GIL
=C2=A0 =C2=A0 PyGILState_Release(gstate);
}

int main() {
=C2=A0 =C2=A0 // Initialize the Python Interpreter
=C2=A0 =C2=A0 Py_Initialize();

=C2=A0 =C2=A0 // Create a vector of threads
=C2=A0 =C2=A0 std::vector<std::thread> threads;

=C2=A0 =C2=A0 // Launch threads
=C2=A0 =C2=A0 for (int i =3D 0; i < 5; ++i) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 threads.push_back(std::thread(thread_function)=
);
=C2=A0 =C2=A0 }

=C2=A0 =C2=A0 // Join threads
=C2=A0 =C2=A0 for (auto& t : threads) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 t.join();
=C2=A0 =C2=A0 }

=C2=A0 =C2=A0 // Finalize the Python Interpreter
=C2=A0 =C2=A0 Py_Finalize();

=C2=A0 =C2=A0 return 0;
}
sunnuntaina 7. hein=C3=A4kuuta 2024 klo 10:24 ip, Tomas Ukkonen <tomas.ukk= onen@protonmail.ch> kirjoitti:

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubun=

tu Linux.

=

/*=C2=A0* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts) =C2=A0*
=C2=A0* g++ python_thread_test.cpp `python3-config --cflags --libs --emb=

ed`

=C2=A0* ./a.out
=C2=A0*
=C2=A0* uname:
=C2=A0* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Li=

nux

=C2=A0*/
=

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>
=

void perform_long_operation() {
=C2=A0 =C2=A0 // Simulate a long-running task
=C2=A0 =C2=A0 std::this_thread::sleep_for(std::chrono::seconds(5));
}
=

void thread_function() {
=C2=A0 =C2=A0 // Ensure this thread has the GIL
=C2=A0 =C2=A0 PyGILState_STATE gstate =3D PyGILState_Ensure();
=

=C2=A0 =C2=A0 // Execute some Python code
=C2=A0 =C2=A0 PyRun_SimpleString("print('Hello from std::thread!')");
=

=C2=A0 =C2=A0 // Release the GIL for long operation
=C2=A0 =C2=A0 Py_BEGIN_ALLOW_THREADS
=C2=A0 =C2=A0 perform_long_operation();
=C2=A0 =C2=A0 Py_END_ALLOW_THREADS
=

=C2=A0 =C2=A0 // Re-acquire the GIL and execute more Python code
=C2=A0 =C2=A0 gstate =3D PyGILState_Ensure();
=C2=A0 =C2=A0 PyRun_SimpleString("print('Thread operation completed!')")=

;

=

=C2=A0 =C2=A0 // Release the GIL
=C2=A0 =C2=A0 PyGILState_Release(gstate);
}
=

int main() {
=C2=A0 =C2=A0 // Initialize the Python Interpreter
=C2=A0 =C2=A0 Py_Initialize();
=

=C2=A0 =C2=A0 // Create a vector of threads
=C2=A0 =C2=A0 std::vector<std::thread> threads;
=

=C2=A0 =C2=A0 // Launch threads
=C2=A0 =C2=A0 for (int i =3D 0; i < 5; ++i) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 threads.push_back(std::thread(thread_functio=

n));

=C2=A0 =C2=A0 }
=

=C2=A0 =C2=A0 // Join threads
=C2=A0 =C2=A0 for (auto& t : threads) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 t.join();
=C2=A0 =C2=A0 }
=

=C2=A0 =C2=A0 // Finalize the Python Interpreter
=C2=A0 =C2=A0 Py_Finalize();
=

=C2=A0 =C2=A0 return 0;
}
=

=

=

Tomas Ukkonen

-----------------------9e29034e00ea96b7d10a477458ea4ed4--

--------cea9a307b89f84c86fe7aee189ffbbfe37df1fbb2efb3b4f35743011c9dd2ff9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wsBzBAEBCAAnBYJmiu86CZCaEYwMt4b97BYhBJbmzyegv3Wo8O5gpZoRjAy3 hv3sAAAGZQgAry3/hOkYMNAsE4FgfSHzeGmxt9xdeejo6X8TYT+6SqxtPxZw cIdIBxG2HodcJ625hz0Ouj0Ft+TZZOhwOrk6uPZUsya0Q1CoMfALl4jpaigh k2jB5alPivARIrTbg0jreVhrXd+HjSmIie0Y3j1e5Vg2v6u+z+hOH2JXhlyx +CFwCvBdKTERIy5HaT2CQNBBBpt/AOUjT2L4v333qOBqcu7BHihN+v5cMQe0 RyzOeA/W7yVpB9d89CedS78PnrCu86Hgky1zisWJ2TBw4gomZcF1hdJnwNuQ LrK3piMdNfCo2Ib08Hc7cVDy58/L7PdUXuUf8QTigw7m7QLWOpJKIQ==
=7W+t
-----END PGP SIGNATURE-----


--------cea9a307b89f84c86fe7aee189ffbbfe37df1fbb2efb3b4f35743011c9dd2ff9--


--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 2024-07-07 20:40, Tomas Ukkonen via Python-list wrote:

Hi

There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).

Updated code:

/*ÿ* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
ÿ*
ÿ* g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
ÿ* ./a.out
ÿ*
ÿ* uname:
ÿ* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
ÿ*/

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void thread_function() {
ÿ ÿ // Ensure this thread has the GIL
ÿ ÿ PyGILState_STATE gstate = PyGILState_Ensure();

ÿ ÿ // Execute some Python code
ÿ ÿ PyRun_SimpleString("print('Hello from std::thread!')");

ÿ ÿ // Release the GIL
ÿ ÿ PyGILState_Release(gstate);
}

int main() {
ÿ ÿ // Initialize the Python Interpreter
ÿ ÿ Py_Initialize();

ÿ ÿ // Create a vector of threads
ÿ ÿ std::vector<std::thread> threads;

ÿ ÿ // Launch threads
ÿ ÿ for (int i = 0; i < 5; ++i) {
ÿ ÿ ÿ ÿ threads.push_back(std::thread(thread_function));
ÿ ÿ }

ÿ ÿ // Join threads
ÿ ÿ for (auto& t : threads) {
ÿ ÿ ÿ ÿ t.join();
ÿ ÿ }

ÿ ÿ // Finalize the Python Interpreter
ÿ ÿ Py_Finalize();

ÿ ÿ return 0;
}
sunnuntaina 7. hein„kuuta 2024 klo 10:24 ip, Tomas Ukkonen <tomas.ukkonen@protonmail.ch> kirjoitti:

Hello
Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.

/*ÿ* This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
ÿ*
ÿ* g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
ÿ* ./a.out
ÿ*
ÿ* uname:
ÿ* Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
ÿ*/

#include <Python.h>
#include <thread>
#include <vector>
#include <iostream>

void perform_long_operation() {
ÿ ÿ // Simulate a long-running task
ÿ ÿ std::this_thread::sleep_for(std::chrono::seconds(5));
}

void thread_function() {
ÿ ÿ // Ensure this thread has the GIL
ÿ ÿ PyGILState_STATE gstate = PyGILState_Ensure();

ÿ ÿ // Execute some Python code
ÿ ÿ PyRun_SimpleString("print('Hello from std::thread!')");

ÿ ÿ // Release the GIL for long operation
ÿ ÿ Py_BEGIN_ALLOW_THREADS
ÿ ÿ perform_long_operation();
ÿ ÿ Py_END_ALLOW_THREADS

ÿ ÿ // Re-acquire the GIL and execute more Python code
ÿ ÿ gstate = PyGILState_Ensure();
ÿ ÿ PyRun_SimpleString("print('Thread operation completed!')");

ÿ ÿ // Release the GIL
ÿ ÿ PyGILState_Release(gstate);
}

int main() {
ÿ ÿ // Initialize the Python Interpreter
ÿ ÿ Py_Initialize();

At this point, there's only one thread (the main thread) and it owns the
GIL.

ÿ ÿ // Create a vector of threads
ÿ ÿ std::vector<std::thread> threads;

ÿ ÿ // Launch threads
ÿ ÿ for (int i = 0; i < 5; ++i) {
ÿ ÿ ÿ ÿ threads.push_back(std::thread(thread_function));
ÿ ÿ }

The threads will each try to acquire and release the GIL, but it's still
owned by the main thread.

ÿ ÿ // Join threads
ÿ ÿ for (auto& t : threads) {
ÿ ÿ ÿ ÿ t.join();
ÿ ÿ }

The main thread is waiting for the sub-threads to finish, and the
threads waiting for the GIL, but the main thread still owns the GIL, so they'll be waiting forever. Deadlock.

ÿ ÿ // Finalize the Python Interpreter
ÿ ÿ Py_Finalize();

ÿ ÿ return 0;
}

--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 7 Jul 2024, at 22:09, Tomas Ukkonen via Python-list <python-list@python=

..org> wrote:

=20
Py_Initialize();

You also need to tell python to init threading.
I think you are missing more python setup code before you can use threads.
Also i think you need to tell python that your thread wants to call into pyt= hon.
But I an not near my dev system to research this for you.

I have code to use python from C++ in my pysvn project.
See the code starting a line 354 in https://sourceforge.net/p/pysvn/code/HEA= D/tree/trunk/pysvn/Extension/Source/pysvn.cpp
That saves the thread state and restores it.

But in my case python creates the threads and I release and acquire the GIL.=


Barry




--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)

On 7 Jul 2024, at 23:21, Barry via Python-list =

<python-list@python.org> wrote:

=20
=20
=20

On 7 Jul 2024, at 22:09, Tomas Ukkonen via Python-list = <python-list@python.org> wrote:
=20
Py_Initialize();

=20
You also need to tell python to init threading.

I'm in front of my dev machine now and checking up on threading.

There is no longer any extra init for threads required.

I think you are missing more python setup code before you can use =

threads.

Also i think you need to tell python that your thread wants to call =

into python.

But I an not near my dev system to research this for you.

You are right to use PyGILState_Ensure()

But as MRAB says the main thread is holding the GIL.

=20
I have code to use python from C++ in my pysvn project.
See the code starting a line 354 in =

https://sourceforge.net/p/pysvn/code/HEAD/tree/trunk/pysvn/Extension/Sourc= e/pysvn.cpp

That saves the thread state and restores it.

You still might find the classes I wrong to manage GIL acquire and =
release interesting.
I have the C++ type system enforcing the rules of acquire and release.
As well as RAII ensuring never to leave a block with the GIL in the =
wrong state.

Barry


--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)