Surrerstry.pl

Entries Projects About

 My multithreading library on pypy-stm

19 April 2018, (Updated: 5 May)

The begin idea was to rewrite basic Python builtin functions in threading and/or multiprocessing way, but I made some tests and in the end I rewrote one standard method and one devised by me. I've choosen operations where multithreading can have best influance on them.
Why I left multiprocessing eventually for this idea? Because on all my tests cost of cloning process was higher then advantages from it. Yes, in long run it could give us better results but, this library is not for that. We create container with data that we have already loaded in RAM, so the bigger process is, the worst multiprocessing will manage with copying process(will need more time). Cost of cloning process will always exceed benefits, because those fundamental functions are not so complicated, and sometimes on one thread they are even finished before processes clone themselves.

If you run this library like script you'll see doctests, and efficiency tests like below:

Results from efficiency tests(2 cores processor):
::: Efficiency TESTS :::

1) .count on list(or tuple)
Generating data...
A) distributed_container: 4.16513299942 seconds
B) standard method: 6.29624915123 seconds

2) .indexes in list(or tuple)
Generating data...
A) distributed_container: 0.333641052246 seconds
B) one thread way: 11.0499639511 seconds

Nevertheless not each method is worth to create its multithreading version, for example method like .reverse is so fast that it doesn't make sense to implement it on many threads...

Additionaly I noticed that in cases when resources(ram, cpu) are overused by other processes this approach seems to be less effective than normal approach. But in general the bigger data we work on, the better results we have in comparison to normal approach.

Samples of usage: (also in doctests):
>>> # 'TESTS_OF:self.count'
>>> from random import randint
>>> input_container = [randint(5,15) for x in range(1000)]
>>> container = distributed_container(input_container, 16)
>>> multithreading_result = container.count(10)
>>> one_thread_result = input_container.count(10)
>>> multithreading_result == one_thread_result
True

>>> # 'TESTS_OF:self.indexes'
>>> input_container = [0,1,2,3,4,5,6,7,8,9,10]*99
>>> container = distributed_container(input_container, 64)
>>> container.indexes(10)
[10, 21, 32, 43, 54, 65, 76, 87, 98, 109, 120, 131, 142, 153, 164, 175, 186, 197, 208, 219, 230, \
241, 252, 263, 274, 285, 296, 307, 318, 329, 340, 351, 362, 373, 384, 395, 406, 417, 428, 439, \
450, 461, 472, 483, 494, 505, 516, 527, 538, 549, 560, 571, 582, 593, 604, 615, 626, 637, 648, \
659, 670, 681, 692, 703, 714, 725, 736, 747, 758, 769, 780, 791, 802, 813, 824, 835, 846, 857, \
868, 879, 890, 901, 912, 923, 934, 945, 956, 967, 978, 989, 1000, 1011, 1022, 1033, 1044, 1055, \
1066, 1077, 1088]

--------------------------------
Update 1, (24 April 2018)
--------------------------------
I added two new methods to my library, and made some refactorization. Below examples of time performance. Examples of usage are in doctests.

Efficiency tests of new methods:
::: Efficiency TESTS :::

   ::Since version 0.2::

3) .remove_all from list
Generating data...
A) distributed_container: 0.261375904083 seconds
B) one thread way: 9.0078458786 seconds

4) .sort list
Generating data...
A) distributed_container: 3.87303900719 seconds
B) one thread way: 9.21541905403 seconds

--------------------------------
Update 2, (1 May 2018)
--------------------------------
Another two methods added to my library namely min and min, they works exactly like builtin min and max.
Additionaly one new feature, now we can define different amount of chunks and threads, if you don't set up amount of threads, script will use amount of threads that corresponds to your amout of cores.
And besides above some small refactorization stuff. Below examples of time performance.
And like always, examples of usage are in doctests.

Efficiency tests of new methods:
::: Efficiency TESTS :::

   ::Since version 0.3::

5) .min
Generating data...
A) distributed_container: 6.65559506416 seconds
B) one thread way: 10.6897859573 seconds

6) .max
A) distributed_container: 6.46050286293 seconds
B) one thread way: 10.6895339489 seconds

--------------------------------
Update 3, (5 May 2018)
--------------------------------
Version 0.4 was released, added keyword arguments to min and max methods, by the way I created internal method to parse keyword arguments that can be easily applicable later.
And beside of this like in each new version, implemented next method, this time bytearray.

Efficiency tests of new methods:
::: Efficiency TESTS :::

   ::Since version 0.4::

7) .bytearray
A) distributed_container: 2.64852404594 seconds
B) one thread way: 3.14809703827 seconds

Source code:
Github Multithreading library on pypy-stm
Onsite distributed_containers.py.zip


Posts



Miscellaneous


Surrerstry.pl © 2014-2017-2018