How to upload and process the CSV file in Django

how to read csv file contents in python django

In this article we will discuss how to upload a csv file and then process the content without storing file on server. One approach could be uploading the file, storing it in upload directory and then reading the file. Another approach could be uploading file and reading it directly from post data without storing it in memory and displaying the data.

We will work with the later approach here.

Uploading CSV file:

First create HTML form to upload the csv file. Use below code for the same.

Important: Do not forget to include enctype="multipart/form-data"  in form.

upload csv in django
Add a URL in URLpatterns.

Create a function in views with the name upload_csv .

Process the CSV file:

In view function, get the file from post data and process it. I used below code in my project.

 

In the above code we are performing below actions:

  • If this is a GET request then render the upload csv html file.
  • If this is a POST request then proceed.
  • First check if file name is not ending with .csv then this is not the valid file. You may implement you own checks as well.
  • Then we check if file is too large. If these tests fail, we return to html form page with appropriate error message. For displaying error/success messages, we are using messages framework. Please import required modules.
  • Then we read the file and split the content by new line character.
  • Iterate over each line and split the line using comma.
  • We are assuming that our csv file have 4 columns of data. We stored the data in a dictionary and then pass the data dictionary to a form.
  • If form is valid then we proceed to save the form and hence creating entry in DB.
  • If form is not valid, or any other error is thrown then we log the error in log file. Read here about logging the errors in log files. This might be useful on live servers where debug is set to false.

Please provide your inputs.

 

(Visited 10,651 times, 1 visits today)

You must read this :

40 thoughts on “How to upload and process the CSV file in Django”

    1. myapp should be the name of your app.(if you have used any other name use that or simply add myapp to list of installed app as specified earlier in settings.py)

  1. Hi there,
    Thanks for this wonderful explanation.
    I want to update/add data in my app as per the fields defined in the model; means I dont want to update the form. Can you help me how should I validate that data and add it directly to database?

    (A quick insight for the project – its an e-commerce kind of platform where you upload the product data in csv file and products get added into your existing database.)

    1. Hi Gauri,
      My apologies as its too late. May be someone else can be benefited by this answer.
      You need to upload the csv and iterate over the rows to create and populate the modelForm, validate it and then save it.
      I will try to write another article focusing on same. Will share the link here once it is done.

  2. Thanks for this guide!

    I’m having an issue when I go to upload the file.
    It is saying “Unable to upload file. UnboundLocalError(“local variable ‘form’ referenced before assignment”,)”.
    Any ideas why this is?

    Thanks!

  3. When i upload the .csv file it’s showing an error.
    Unable to upload file. IndexError(‘list index out of range’,)
    Please Help!

    Also, can you upload an example code to GitHub and share the link. It’d be a great help. Thank You.

    1. Hi Saurabh,
      Thanks for stopping by. Unfortunately I don’t have the source code now. I will try to create a new project and upload it on github.
      Meanwhile can you please paste the complete stack trace of error. I will certainly help you mitigate the issue.
      As per my code, it assumes that your csv have four columns. Are there 4 columns (i.e. 4 values separated by 3 commas) in your csv? either create a 4 column csv or update the code to get only columns which are in range.

        1. Hi Saurabh,
          Please make it a habit to share the error stack trace in pastebin.
          Its easier to find out the line number and file where problem is. Now its too hard to find the exact file and line.

        2. Please change

          except Exception as e:
          logging.getLogger(“error_logger”).error(form.errors.as_json())
          pass

          to

          except Exception as e:
          logging.getLogger(“error_logger”).error(repr(e))
          pass

  4. I a little bit modified the code for my setup and make it work but some how it’s only gathering only first letter of each line from the csv file and writing to the database .What may I be doing wrong ?

  5. Hey, I have implemented the csv upload but my loop only reads the first row and it stops. I am not sure why for loop not going the csv file.

  6. Why the wheel reinvention? Don’t parse CSV manually, there is a csv module in the standard library:

    import csv, io

    and to parse into ready-to-use dictionaries:

    with io.TextIOWrapper(request.FILES[“csv_file”].file, encoding=’utf8′, newline=”) as csvfile:
    reader = csv.DictReader(csvfile, fields=(‘name’, ‘start_date_time’, ‘end_date_time’, ‘notes’))
    for data_dict in reader:

    (a gist of the same: https://gist.github.com/mjpieters/d92ad837a6262cc2699b20d44d295c62)

    The reader can be further configured to produce an error on extra columns, or silently drop them, etc.

  7. Am new to python , just need a small help
    I understand code only to some extent , but not sure how to run it on my machine, I have django and python installed properly. Not sure how and where to save above 2 files.
    Getting below errors
    url(r’^upload/csv/$’, views.upload_csv, name=’upload_csv’),
    NameError: name ‘url’ is not defined
    and
    NameError: name ‘views’ is not defined

  8. Hi,
    I am trying to upload .csv file, but it’s neither uploading file nor showing any kind of error.
    Even if I am uploading other type of file it’s not showing any error.

    1. {“system_id”: [{“message”: “Enter a whole number.”, “code”: “invalid”}], “ext”: [{“message”: “Enter a whole number.”, “code”: “invalid”}]}
      {“ext”: [{“message”: “Enter a whole number.”, “code”: “invalid”}]}
      {“ext”: [{“message”: “Enter a whole number.”, “code”: “invalid”}]}
      {“ext”: [{“message”: “Enter a whole number.”, “code”: “invalid”}]}
      {“ext”: [{“message”: “Enter a whole number.”, “code”: “invalid”}]}
      Unable to upload file. IndexError(‘list index out of range’,)
      this is the error am getting

    2. after uploading file it’s not parsing or encoding special character for example. ‘@’ to ‘+AEA-‘

  9. Hi

    I am new to Django. Can you give the code for EventsForm which is mentioned in the code above(views.py)

    Thanks
    Srinivasa

    1. @srinivasa, Sorry I don’t have the source code with me now.
      It would be a ModelForm with all fields in your DB table.

Leave a Reply

Your email address will not be published. Required fields are marked *