python - How to use celery for inserting data to mongodb using mongoengine -

-

I am trying to use celery to incorporate large data into my mongodb, but problem mangodb in concurrency If I send more than one work to the vegetable part of the data at one time, then it will be inserted into mongodb and there will be no other data. I think this is because MongoDob locks the database on the functioning of inserting data, but I need a solution to be able to send several functions of the same type to enter data into the database. As if the database is locked when it is waiting to unlock it. Here is a part of my code: 

  @ celery.task (name = 'celery_tasks.add_book_product') def add_book_product (product_dict, store_id): connect (DefaultConfig.MONGODB_DB, host = DefaultConfig.MONGODB_HOST ) Try store_obj = Store.objects.get (pk = store_id): book = books.objects.get (pk = product_dict ['rawbook']) Try: product_obj = product. Absence. (Store = store_obj, related_book = book, kind = 'book') print (found for "product {} store {}". Format (product_obj.id, store_obj.id)) product_obj.count = int (product_dict ['count ']) Product_obj.buy_price = int (product_dict [product_obj.sell_price = int (product_dict [' sell_book ']) except product_obj.save () (DoesNotExist, ValidationError): product_obj = product (store = store_obj, related_book = book, kind = 'Book', count = int (product_dict ['count']), buy_price = int (product_dict ['buy_book']), sell_price = int (product_dict ['sell_book']), name = book.name_fa) product_obj.save () Print ("Adding books to store oz ...") store_obj.products.append (product_obj) store_ob j.save () print ("to store oboes To engage Itaben ") return" product saved for {} stores {} "). Excluding the format (product_obj.id, store_obj.id) (DoesNotExist, ValidationError): traceback.print_exc () Returns "Raw book {{PRODUCT WITH A raw book}} does not exist." Format (product_dict ['rawbook'])    
 
 
 Using multi-processing by default In celery is done for concurrent performance of the work. But there are two ways to ensure that only one task is executed at any one time.  
  Solution 1:   
 When you call a celery employee  
  Celery - your_app worker - l info   
 The default concurrency is the number of the core in your machine. So if you start an employee like this then  
  celery -e your_app worker -l info -c 1   
 this is just a time Work runs. If you have some other tasks to execute, then you can start a new queue and allocate a worker to do this.  
  Solution 2:   
 It's a bit complicated, you have to use lock in your work, something like that.  
  if acquire_lock (): try: #do something finally: release_lock () Return   
 You can read more about this.

Comments

Post a Comment

Popular posts from this blog

php - PDO bindParam() fatal error -

-
I am trying to learn PDO, now I have written this small code, but it gives me a serious error: Fatal error: Call a member function at a non-object dot () ... $ con = new mysqli ("127.0.0.1", "root", "", "csvdangercheck"); $ Query = $ con- & gt; ("Id,` var1`, `var2`) values ​​(" id,: var1,: var2) ";); $ query- & gt; bypass pattern (': id', $ id); $ query- & gt; BindParam (': var1', $ VAL1); $ Query- & gt; bindParam (': var2', $ val2); $ Query-> Executed (); I Trying to use print_r ($ con-> errorInfo ()) but Fatal error: Undefined method mysqli :: errorInfo ( ) ... can someone tell me what am I missing? Flaffeh said, you have PDO Missing Mysqy, try it: $ con = new PDO ('mysql: host = 127.0.0.1; dbname = csvdangercheck', 'root', '');
Read more

php - How can I cram 6+31 numeric characters into 22 alphanumeric characters? -

-
I have a 6 digit number and a 31-digit number (such as "234536" and "201103231043330478311223582826") that I The API needs to be crawled in the same 22-character alphanumeric field that uses PHP. I tried to convert each one to 32 (using custom code could not handle the base_convert () large number) and to connect with a single-character delimiter, but It only gives me 26 characters This is a restore API, so the letters should be URI-protected. I would like to do a database table cross with two references without reference to another reference value, if possible, any suggestions? Instead use a radix of 62. This will give you 22 characters for the top, 3.35 for the pre and 17.3 for the next. & gt; & Gt; & Gt; Math.log (10 ** 6) /math.log (62) 3.3474826039165504 & gt; & Gt; & Gt; Math.log (10 ** 31) /math.log (62) 17.295326786902177
Read more

logging - How can I log both the Request.InputStream and Response.OutputStream traffic in my ASP.NET MVC3 Application for specific Actions? -

-
For a specific set of tasks, I need to log in to the incoming request inputstream as well as outgoing response. is. I am using an ActionFilterAttribute for this and overriding the OnActionExecuted and OnResultExecuted methods. So I'm starting with this idea ... Public category ActionLoggerAttribute: ActionFilterArrbit {Public Override Zero On-Offed Execution (ActionActioned Contact Filter Filter) {base.OnActionAxecuted (FilterContext); HttpRequestBase request = filterContext.HttpContext.Request; // TODO: Log the request. InputStream} Public Override Zero OnResultExecuted (ResultExecutedContext FilterContext) {base.OnResultExecuted (filterContext); HTTPRPSPointBase Response = FilterContacts Hpptex response; // TODO: Log on Response.OutputStream}} Ideally I'll hook it up with enterprise library logging because I'm already using it for error logging. Am I reaching the input and output stream at the right time? Can I easily use the Enterprise Library to log i...
Read more