[SOLVED] Cannot send more than 6 variables w/Arduino

Hi!
I have a garden soil temperature and moisture sensor project with a total of 17 sensors (8 temp, 7 moisture, 1 light and 1 humidity) connected to an Arduino Mega 2560 R3. Components: DS18B20’s, VH400’s, Adafruit CC3000 WiFi breakout, Adafruit microSD card breakout, Spark Fun DeadOn RTC, DHT22, and CdS photoresistor.

While learning to use Ubidots, I have been only sending one variable at a time: Average temp, average moisture, raw temp from one sensor, raw moisture from one sensor, humidity, and light.

I have been able to send up to 6 of those variables in any combination to Ubidots without any problems, but when I try sending a 7th variable, the Arduino program resets itself and starts over (without the programmed delay or disconnecting WiFi). I’ve also tried sending the data using the “collections” API, but the Arduino program just hangs. Any suggestions?

I’ve only seen one other topic here that mentions a similar problem (http://community.ubidots.com/t/cant-get-collections-to-work-with-arduino/217), but it’s almost a month old with no reply. If I’m to eventually add all the sensors (and be a paying customer) I’d first like to know that this is a solvable problem.

Thanks!

Here’s the code. Some of it may have been greyed out during troubleshooting:
Garden_Project_UbiDots_forum_010315.ino (18.0 KB)

Hello carl! how are you?
Well i saw your code and i have some reason for your problem

  1. The memory of the ATmel is just 2.5mb and your code is very long
  2. You are using a single post value for every code and you can do it with only one function like this:

http://ubidots.com/docs/api/v1_6/collections/post_values.html#post-api-v1-6-collections-values

Where in your code is like this:

   client.println(F("POST /api/v1.6/collections/values HTTP/1.1"));
   client.println(F("Content-Type: application/json"));
   client.print(F("Content-Length: "));
   client.println(stringLength);
   client.print(F("X-Auth-Token: "));
   client.println(token);
   client.println(F("Host: things.ubidots.com"));
   client.println();
   client.println(webData);
   client.println();

where webData is like:

   String webData = "[{"variable": "527e656f73d1513793b2d05123", "value":2}, {"variable": "527e656f73d1513793b2d059", "value":23, "variable": "527e656f71231513793b2d051", "value":278}]"
String stringLength = webData.length();

Note: the collections is like [{“variable”: “id_variable_1”, “value”: value_variable_1}, {“variable”: “id_variable_2”, “value”: value_variable_2}, {“variable”: “id_variable_3”, “value”: value_variable_3}]

Best regards,
Metavix

Hi Metavix. Thanks a bunch for the reply.

It’s big, but only using about 18% of the memory. :smiley:

I’ve assembled the POST code as you said, like this:

// Send webData to Ubidots
Adafruit_CC3000_Client client = cc3000.connectTCP(ip, 80);
if (client.connected())
{
Serial.println(F(“Connected and sending webData to Ubidots!”));
client.println(F(“POST /api/v1.6/collections/values HTTP/1.1”));
Serial.println(F(“POST /api/v1.6/collections/values HTTP/1.1”));
client.println(F(“Content-Type: application/json”));
Serial.println(F(“Content-Type: application/json”));
client.print(F("Content-Length: "));
Serial.print(F("Content-Length: "));
client.println(stringLength);
Serial.println(stringLength);
client.print(F("X-Auth-Token: "));
Serial.print(F("X-Auth-Token: "));
client.println(token);
Serial.println(token);
client.println(F(“Host: things.ubidots.com”));
Serial.println(F(“Host: things.ubidots.com”));
client.println();
Serial.println();
client.println(webData);
Serial.println(webData);
client.println();
Serial.println();
Serial.println(F(“WebData Succefully sent to Ubidots!”));
}
else
{
Serial.println(F(“Connection for webData failed.”));
return;
}

But am having trouble with the webData string assembly:

Example A
String webData = “[{"variable": “+ String(idvariable1) +”, "value": “+ String(AirTemp) +”}, {"variable": “+ String(idvariable2) +”, "value": “+ String(AvgTemp) +”}, {"variable": “+ String(idvariable3) +”, "value": “+ String(AvgMoist) +”}, {"variable": “+ String(idvariable4) +”, "value": “+ String(Temp5) +”}, {"variable": “+ String(idvariable5) +”, "value": “+ String(analogRead(7)) +”}, {"variable": “+ String(idvariable6) +”, "value": “+ String(photocellReading) +”}, {"variable": “+ String(idvariable7) +”, "value": “+ String(h) +”}]”;

Example B
String webData = “[{“variable”: “+ String(idvariable1) +”, “value”: “+ String(AirTemp) +”}, {“variable”: “+ String(idvariable2) +”, “value”: “+ String(AvgTemp) +”}, {“variable”: “+ String(idvariable3) +”, “value”: “+ String(AvgMoist) +”}, {“variable”: “+ String(idvariable4) +”, “value”: “+ String(Temp5) +”}, {“variable”: “+ String(idvariable5) +”, “value”: “+ String(analogRead(7)) +”}, {“variable”: “+ String(idvariable6) +”, “value”: “+ String(photocellReading) +”}, {“variable”: “+ String(idvariable7) +”, “value”: “+ String(h) +”}]”;

Example C
String webData = “[{“variable”: (idvariable1), “value”: (AirTemp)}, {“variable”: (idvariable2), “value”: (AvgTemp)},{“variable”: (idvariable3), “value”: (AvgMoist)}, {“variable”: (idvariable4), “value”: (Temp5)}, {“variable”: (idvariable5), “value”: (analogRead(7))}, {“variable”: (idvariable6), “value”: (photocellReading)}, {“variable”: (idvariable7), “value”: (h)}]”;

Example D
String webData = “[{“variable”: “idvariable1”, “value”: “AirTemp”}, {“variable”: “idvariable2”, “value”: “AvgTemp”}, {“variable”: “idvariable3”, “value”: “AvgMoist”)}, {“variable”: “idvariable4”, “value”: “Temp5”}, {“variable”: “idvariable5”, “value”: “analogRead(7)”}, {“variable”: “idvariable6”, “value”: “photocellReading”}, {“variable”: “idvariable7”, “value”: “h”}]”;

Example E
String webData = “[{“variable”: “(idvariable1)”, “value”: “(AirTemp)”}, {“variable”: “(idvariable2)”, “value”: “(AvgTemp)”}, {“variable”: “(idvariable3)”, “value”: “(AvgMoist)”)}, {“variable”: “(idvariable4)”, “value”: “(Temp5)”}, {“variable”: “(idvariable5)”, “value”: “(analogRead(7))”}, {“variable”: “(idvariable6)”, “value”: “(photocellReading)”}, {“variable”: “(idvariable7)”, “value”: “(h)”}]”;

stringLength = webData.length();

Example A will compile, but when the program runs it just hangs at “Host: things.ubidots.com”, not even going to the ‘else’ statement.

Here’s what the serial output looks like:

SD OK
Date & Time, Light, %Humidity, AirTemp, Bed 1, , Bed 3, , Bed 5, , Bed 7, , Bed 9, , Bed 11/12, , Tomatoes, , Average Temp, , Average Moisture,
Initializing WiFi…
WiFi Success!
Attempting to connect to HOME-7A02
WiFi Connected! 1
Requesting DHCP. 1
.
DHCP complete. 1
things.ubidots.com → 50.23.124.68
Date-Time:
08.01.2015-16:16:44
% Humidity: 50.90
Light sensor = 438 - DayLight
AirTemp = 21.87
Temp1: 21.87 Moisture1: 91
Temp3: 22.00 Moisture3: 83
Temp5: 21.81 Moisture5: 72
Temp7: 21.69 Moisture7: 81
Temp9: 21.25 Moisture9: 96
Temp11: 21.81 Moisture11: 103
TempT: 22.75 MoistureT: 136
Average Temp: 21 Average Moisture: 87

08.01.2015-16:16:45,438,50.90,21.88,21.88,90,22.00,82,21.81,70,21.69,81,21.25,97,21.81,102,22.75,136,21,87
Connected and sending webData to Ubidots!
POST /api/v1.6/collections/values HTTP/1.1
Content-Type: application/json
Content-Length: 126
X-Auth-Token: xxxxxx
Host: things.ubidots.com

Examples B, C, D, and E are variations on your example, but the IDE won’t compile any of them and gives the error “expected ‘,’ or ‘;’ before ‘variable’”

I assume Example A is correct JSON payload structure for Arduino?

Also, how do you post on here so it looks like the code is in the IDE (like in your reply)?

–Carl

@Carl

I tried this code with a CC3000 Shield from Adafruit and an Arduino Uno. The code easily allows you to add 3, 4, 5 and up to 6 variables to your request; just move the “/" and "/” lines in the “save_values” function to uncomment the needed variables:

**** As you can see in the code, your “Example A” is the right way to send a collection"****

/*************************************************** 
  Based on the example from Adafruit's CC3000 Library
 ****************************************************/
 
 /*
This example was built to find out how many Ubidots variables can be updated in a single request using the CC3000. 
*/

#include <Adafruit_CC3000.h>
#include <ccspi.h>
#include <SPI.h>
#include <string.h>
#include "utility/debug.h"

// These are the interrupt and control pins
#define ADAFRUIT_CC3000_IRQ   3  // MUST be an interrupt pin!
// These can be any two pins
#define ADAFRUIT_CC3000_VBAT  5
#define ADAFRUIT_CC3000_CS    10
// Use hardware SPI for the remaining pins
// On an UNO, SCK = 13, MISO = 12, and MOSI = 11
Adafruit_CC3000 cc3000 = Adafruit_CC3000(ADAFRUIT_CC3000_CS, ADAFRUIT_CC3000_IRQ, ADAFRUIT_CC3000_VBAT,
                                         SPI_CLOCK_DIVIDER); // you can change this clock speed

#define WLAN_SSID       "OpenWRT"           // cannot be longer than 32 characters!
#define WLAN_PASS       "123456789"
// Security can be WLAN_SEC_UNSEC, WLAN_SEC_WEP, WLAN_SEC_WPA or WLAN_SEC_WPA2
#define WLAN_SECURITY   WLAN_SEC_WPA2

#define IDLE_TIMEOUT_MS  3000      // Amount of time to wait (in milliseconds) with no data 
                                   // received before closing the connection.  If you know the server
                                   // you're accessing is quick to respond, you can reduce this value.
                                   

////////////////////////////////////// Ubidots parameters
#define WEBSITE "things.ubidots.com"
#define URL "/api/v1.6/collections/values/?token=BDHq8DFi9dcaake6ABF5fkvJk2EWF3&force=true"
#define idvariable1 "568a84237625420e53b118d9"
#define idvariable2 "569133c976254212f0064733"
#define idvariable3 "569133cd762542111c03f057"
#define idvariable4 "569133d2762542111c03f06d"
#define idvariable5 "569133d776254212673e7a19"
#define idvariable6 "569133de762542111c03f0b6"

char payload[360];  // Reserve a char to store the data to send. Account for ~60 bytes per variable. 
char le[4];

/**************************************************************************/
/*!
    @brief  Sets up the HW and the CC3000 module (called automatically
            on startup)
*/
/**************************************************************************/

uint32_t ip;

void setup(void)
{
  Serial.begin(115200);
  Serial.println(F("Hello, CC3000!\n")); 

  Serial.print("Free RAM: "); Serial.println(getFreeRam(), DEC);
  
  /* Initialise the module */
  Serial.println(F("\nInitializing..."));
  if (!cc3000.begin())
  {
    Serial.println(F("Couldn't begin()! Check your wiring?"));
    while(1);
  }
  
  // Optional SSID scan
  // listSSIDResults();
  
  Serial.print(F("\nAttempting to connect to ")); Serial.println(WLAN_SSID);
  if (!cc3000.connectToAP(WLAN_SSID, WLAN_PASS, WLAN_SECURITY)) {
    Serial.println(F("Failed!"));
    while(1);
  }
   
  Serial.println(F("Connected!"));
  
  /* Wait for DHCP to complete */
  Serial.println(F("Request DHCP"));
  while (!cc3000.checkDHCP())
  {
    delay(100); // ToDo: Insert a DHCP timeout!
  }  

  /* Display the IP address DNS, Gateway, etc. */  
  while (! displayConnectionDetails()) {
    delay(1000);
  }
  
  ip = 0;
  // Try looking up the website's IP address
  Serial.print(WEBSITE); Serial.print(F(" -> "));
  while (ip == 0) {
    if (! cc3000.getHostByName(WEBSITE, &ip)) {
      Serial.println(F("Couldn't resolve!"));
    }
    delay(500);
  }

  cc3000.printIPdotsRev(ip);


}

void loop(void)
{



  save_values(analogRead(A0), analogRead(A1), analogRead(A2), analogRead(A3), analogRead(A4), analogRead(A5));
  delay(4000);
  
}

void save_values(int val0, int val1, int val2, int val3, int val4, int val5){

    Serial.println("Free RAM: "); Serial.println(getFreeRam(), DEC);
    // Prepare payload to send to server - using "sprintf" instead of String concatenation to avoid memory issues
    
    Serial.println(F("Setting up payload..."));

    sprintf(payload,"%s", "[");
    sprintf(payload,"%s%s", payload, "{\"variable\":\"" idvariable1 "\",\"value\":");
    sprintf(payload,"%s%d", payload, val0);
/*
    sprintf(payload,"%s%s", payload, "},{\"variable\":\"" idvariable2 "\",\"value\":");
    sprintf(payload,"%s%d", payload, val1);
    sprintf(payload,"%s%s", payload, "},{\"variable\":\"" idvariable3 "\",\"value\":");
    sprintf(payload,"%s%d", payload, val2);
    sprintf(payload,"%s%s", payload, "},{\"variable\":\"" idvariable4 "\",\"value\":");
    sprintf(payload,"%s%d", payload, val3);
    sprintf(payload,"%s%s", payload, "},{\"variable\":\"" idvariable5 "\",\"value\":");
    sprintf(payload,"%s%d", payload, val4);
    sprintf(payload,"%s%s", payload, "},{\"variable\":\"" idvariable6 "\",\"value\":");
    sprintf(payload,"%s%d", payload, val5);
*/
    sprintf(payload,"%s%s", payload, "}]");

    // Get length of the entire payload
    sprintf(le,"%d", strlen(payload));
    
    Serial.println(F("Done! Payload: "));
    Serial.println(payload);
    Serial.println(F("Length: "));
    Serial.println(le);
  

    Serial.println(F("Connecting to Ubidots..."));
    
    Adafruit_CC3000_Client www = cc3000.connectTCP(ip, 80);
    
    if (www.connected()) {
      Serial.println(F("Making request..."));

      Serial.print(F("POST "));
      Serial.print(URL);
      Serial.print(F(" HTTP/1.1\r\n"));
      Serial.print(F("Content-Type: application/json\r\n"));
      Serial.print(F("Connection: close\r\n"));
      Serial.print(F("Host: "));
      Serial.print(WEBSITE); 
      Serial.print(F("\r\n"));
      Serial.print(F("Content-Length: "));
      Serial.print(le);
      Serial.print(F("\r\n"));
      Serial.print(F("\r\n"));
      Serial.print(payload);
      Serial.print(F("\r\n"));
      Serial.println();
      
      www.fastrprint(F("POST "));
      www.fastrprint(URL);
      www.fastrprint(F(" HTTP/1.1\r\n"));
      www.fastrprint(F("Content-Type: application/json\r\n"));
      www.fastrprint(F("Connection: close\r\n"));
      www.fastrprint(F("Host: "));
      www.fastrprint(WEBSITE); 
      www.fastrprint(F("\r\n"));
      www.fastrprint(F("Content-Length: "));
      www.fastrprint(le);
      www.fastrprint(F("\r\n"));
      www.fastrprint(F("\r\n"));
      www.fastrprint(payload);
      www.fastrprint(F("\r\n"));
      www.println();
    } else {
      Serial.println(F("Connection failed"));    
      return;
    }

  Serial.println(F("-------------------------------------"));
  
  /* Read data until either the connection is closed, or the idle timeout is reached. */ 

  unsigned long lastRead = millis();
  while (www.connected() && (millis() - lastRead < IDLE_TIMEOUT_MS)) {
    while (www.available()) {
      char c = www.read();
      Serial.print(c);
      lastRead = millis();
    }
  }
  www.close();
  Serial.println(F("-------------------------------------"));
  
  /* You need to make sure to clean up after yourself or the CC3000 can freak out */
  /* the next time your try to connect ... */
  
  memset(payload, 0, sizeof(payload)); 

}

/**************************************************************************/
/*!
    @brief  Tries to read the IP address and other connection details
*/
/**************************************************************************/
bool displayConnectionDetails(void)
{
  uint32_t ipAddress, netmask, gateway, dhcpserv, dnsserv;
  
  if(!cc3000.getIPAddress(&ipAddress, &netmask, &gateway, &dhcpserv, &dnsserv))
  {
    Serial.println(F("Unable to retrieve the IP Address!\r\n"));
    return false;
  }
  else
  {
    Serial.print(F("\nIP Addr: ")); cc3000.printIPdotsRev(ipAddress);
    Serial.print(F("\nNetmask: ")); cc3000.printIPdotsRev(netmask);
    Serial.print(F("\nGateway: ")); cc3000.printIPdotsRev(gateway);
    Serial.print(F("\nDHCPsrv: ")); cc3000.printIPdotsRev(dhcpserv);
    Serial.print(F("\nDNSserv: ")); cc3000.printIPdotsRev(dnsserv);
    Serial.println();
    return true;
  }
}

After testing, I found that the CC3000 starts to block when sending/receiving larger amounts of data. This is also mentioned in Adafruits forums (http://forums.adafruit.com/viewtopic.php?f=31&p=273395):

As far as fixing this unfortunately I don’t think there’s any easy solution. You could bump up the size of the RX buffer by changing the CC3000_MINIMAL_RX_SIZE define in cc3000_common.h, but it will take more memory from your sketch. It’s annoying that the code halts when large data is received but it prevents an overflow that would cause weird, undefined behavior.

If you have control over what’s sending data to the Arduino, I would try to keep the max data it sends below ~90 bytes (i.e. break things up into a lot of smaller packets). If you don’t have control of it and need to receive a large amount of data you might look at using a mega or due–the due in particular has a lot of memory and could increase the buffer quite a bit (perhaps 20-50kb or more).

tdicola

Posts: 976
Joined: Thu Oct 17, 2013 9:11 pm

So my advise would be to:

  1. Try this code with the Mega to find the limit of variables that can be sent at the same time.

  2. Modify the “save_values” function to receive variable IDs and the amount of supported values. For example, if you could send only 3 values at the same time, the function would look like:

    void save_values(String idvariable1, String idvariable2, String idvariable3, int val1, int val2, int val3){ …

  3. In the main loop, call this function as many times as needed, for example:

    save_values( idvariable1, idvariable2, idvariable3, analogRead(A0), analogRead(A1), analogRead(A2));
    save_values( idvariable4, idvariable5, idvariable5, analogRead(A3), analogRead(A4), analogRead(A5));

Hello Carl,

If you still encounter limitations with the CC3000 and the above does not work, we can just deploy a specific endpoint for you at Ubidots.

So instead of “/api/v1.6/collections/values” you would use a special URL like “/carl/garden/” and we can get your data separated by commas (123,34,43,23,53,42) and then we would process it and send it to your account. This would require some dev time from our end, but it’s probably the simplest solution (moving the problem to the cloud instead of dealing with the CC3000).

By the way, you can skip sending the average values from the device as these can be computed in the cloud. Just add a new variable and then select in the type of variable “Derived”. This will present you with a box to fill in your formula.

Looking forward!

Hi guys,

Thanks for the replies and I apologize for the delay in my response (been busy at work).

As hackmed says, there is an unfortunate known buffer limitation with the CC3000 and it freaks out with strings of ~100 bytes. Seems like it was designed for sending data from a few sensors and not good for large projects.

Here are a few other links besides the one hackmed posted referring to the problem:
https://e2e.ti.com/support/wireless_connectivity/simplelink_wifi_cc3000/f/851/t/321924
https://e2e.ti.com/support/wireless_connectivity/simplelink_wifi_cc3000/f/851/t/312857

hackmed: Thanks for the code. I’m already able to send up to six variables, but it stumbles when I send seven. Does your method work with a lot of variables? I’ll eventually have 16-17. I tried a few other tricks I found on the net, kinda like you’ve shown, by converting strings/floats to char, and sending the data via fastrprint and letting Ubidots concatenate it, but the Ubidots API rejected it and continued to give me 404 and 400 errors. Any other tricks may be out of my coding ability.

Admin: Thanks very much for the offer. Since this is a CC3000 problem, do you think it’ll still choke on 16-17 variables? how would we test it? Also, heard of any other Arduino friendly shields/breakout boards that can handle this kind of data transfer?

Another work around may be to average the temperature and moisture values of all the beds and send them as individual variables. That would only be five values (AvgTemp, AvgMoisture, AirTemp, Humidity and Light).

–Carl

@Carl, it should work with more variables if you call the method several times. If it worked for six, then you can make 3 separate requests to update a total of 18 variables.

Hi @Carl, we have deployed a test endpoint for you so you can send data in TCP mode under this address:

translate.ubidots.com
Port: 9000

That is, just open a socket pointing to that url on port 9000 and stream comma separated values following this format:

Token,Datasource-Name,Var1-Name,Var1-Value,Var2-Name,Var2-Value,Var3-Name,Var3-Value,Var4-Name,Var4-Value,Var5-Name,Var5-Value,Var6-Name,Var6-Value,Var7-Name,Var7-Value,Var8-Name,Var8-Value …etc.

For example, this payload:

“xxxxxxx-your-token-xxxxxxx,Plant-1,Temp,32,Hum,52,var4,2414,var5,124,var6,412,var7,3434,var8,214,var45,533,var9,3423”

Would create this data in your account:

The service is not responding anything right now, so just close the connection yourself without waiting for a response from the server.

Let us know if your need any help modifying your code to test this. Depending on your variables’ names, the payload would be larger or shorter. In any case, rememeber the 100 byte limit!

1 Like

@Carl BTW I just published a new library for the CC3000, hope this helps

cya