Publish with timestamp fails on variables that have a callback

I am trying to publish values with a timestamp. For some variables this works without problem, but with others the value is never received. The ones that fail all have callbacks set. If I omit the timestamp then everything works as expected.

For example, with timestamps, this fails:

11:08:54.063 → Sent /v1.6/devices/xxxxxx,{“btn-open-1”:{“value”:0.00,“timestamp”:1579720131000}}
but this works:
11:08:54.299 → Sent /v1.6/devices/xxxxxx,{“door-state”:{“value”:3.00,“timestamp”:1579720131200}}

Without timestamps, they all work:

11:13:42.230 → Sent /v1.6/devices/xxxxxx,{“btn-close”:{“value”:0.00}}
11:13:42.466 → Sent /v1.6/devices/xxxxxx,{“door-state”:{“value”:1.00}}

When I look at the Device page, I can see that the variable is or is not being updated. The timestamp is correct. I tried sending these 2 variables in either order, but the same one always fails. Why would one work and the other not?

The variables that fail are used as switches, and thus have callbacks. This looks like a bug to me.

Thank you,
Will

Hi there, I tried to reproduce your issue but at my side everything works as expected, please refer to the gif below where I run two mqtt instances, the first one publishes data and the second one implements a subscription with a callback:

mqtt tests

Notice that in the second one I am able to receive both published values and timestamps.

Now, I suspect that the issue may be at PubSubclient, the library sets as max number of packet payload size 128 bytes in its header file, maybe changing it to 512 solve your problem.

Let me know how it goes

I tried increasing the max packet size as you suggested. It did not make any difference. I added code to show both the length of the topic and payload, and also display the return code from the Publish. The combined lengths of topic and payload are less than 100 bytes. The return code from Publish is always 1. The length of the data being sent is actually more for the items that are working.

This one fails:

04:45:23.319 → Sent: /v1.6/devices/ddddd,{“btn-open-1”:{“value”:0.00,“timestamp”:1579783521000}} len=88, rc=1

This one works:

04:45:23.590 → Sent: /v1.6/devices/ddddd,{“door-state”:{“value”:3.00,“timestamp”:1579783521200}} len=88, rc=1

In this case the length of the data being sent is identical (88 bytes)

If I remove the timestamp from the first one, it now works:

04:50:03.045 → Sent: /v1.6/devices/ddddd,{“btn-open-1”:{“value”:0.00}} len=62, rc=1

Looks like it is a timing issue. I added code to reset one variable when the switch for another variable is clicked. When I do this the publish for the variable not receiving the callback always works.

It looks like if I send the value too soon after the callback is received, it is ignored. If I wait a few seconds then it works. That’s why you don’t see it when you use mosquito, you can’t send the value back fast enough to hit the problem.

Hi there, I have just tested with Pubsubclient and at my side, with values spaced in 250 ms (the max refresh rate allowed by Ubidots) I can retrieve properly all the values in a subscribed topic, please reference the gif below where you will find a small test sending 10 dots:

I am not sure where your issue is as I do not know the details of your code routine, but I attach the script that I tested so you can use it for additional throubleshoot at your side

/****************************************
 * Include Libraries
 ****************************************/
#include "UbidotsESPMQTT.h"

/****************************************
 * Define Constants
 ****************************************/
#define TOKEN "...." // Your Ubidots TOKEN
#define WIFINAME "..." //Your SSID
#define WIFIPASS "...." // Your Wifi Pass

Ubidots client(TOKEN);

/****************************************
 * Auxiliar Functions
 ****************************************/

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i=0;i<length;i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

/****************************************
 * Main Functions
 ****************************************/

void setup() {
  // put your setup code here, to run once:
  client.ubidotsSetBroker("business.api.ubidots.com"); // Sets the broker properly for the business account
  client.setDebug(false); // Pass a true or false bool value to activate debug messages
  Serial.begin(115200);
  client.wifiConnection(WIFINAME, WIFIPASS);
  client.begin(callback);
  client.ubidotsSubscribe("weather-station","temperature"); //Insert the dataSource and Variable's Labels
  }

void loop() {
  // put your main code here, to run repeatedly:
  if(!client.connected()){
      client.reconnect();
      client.ubidotsSubscribe("weather-station","temperature"); //Insert the dataSource and Variable's Labels
      }
  client.loop();
  }

All the best

I am not having problems publishing to non-callback variables, or to callback variables when there is no callback. The problem ONLY occurs when there is a callback, so you have to have a dashboard with a switch (or something) that triggers the callback first, then respond with a new value (in this case to reset the switch to zero or “off”). If this publish is done without a timestamp, it works every time. Add a timestamp to the packet, and it fails.

Thanks!
Will

I do not quite understand your setup, may you please clarify to me some doubts about your setup so I can give you additional troubleshoot hints?

The callback that you are referring to is the one of the pubsubclient? or which one? In the previous example that I shared with, the variable had a callback in the arduino script, because that is the way to handle variable updates.

In this, I suppose that there is pubsubclient callback, but I do not quite understand how are you resetting the switch, do you have an Ubidots event? or is it a physical switch connected to your device?

I have not been able to reproduce this, I have sent in my tests several packets with both timestamps and context, and always get all the information. May you share a video of the issue and also if possible a test script to reproduce the problem?

All the best

I can send packets to ubidots when there is no callback without a problem.

I am using PubSubClient and MQTT. I can provide the code if you really need to see it but I do not wish to post it here.

I have some Switch widgets on a dashboard. The Switch widget is not momentary so I send a zero value back when the callback is received (about 250ms later) to reset the switch to “off”. If I send this value with a timestamp it fails. If I send it without a timestamp it works. I tried a trick where I send the “reset” packet from the callback for another switch variable, and if I wait a bit before triggering this it will work. This indicates to me that there is a timing issue where a packet is received with a timestamp for a variable that has a callback and was updated within some time period.

Packets are never sent faster than at 250 ms intervals, as I am using a queue (per your suggestion in another thread). I even tried 300 and 500 ms intervals, but it still fails.

If I omit the timestamp from the packet it works every time. If I send packets with timestamps for variables that do not have callbacks these do not fail.

So the timing is:

  1. switch widget triggers value change
  2. callback is received with the new value
  3. 250ms later a zero value is sent (with a timestamp) for the same variable
  4. the variable is NOT updated by Ubidots
    Simply remove the timestamp and it works. Every time.

I have put quite a bit of effort into tracking this down. I hope you can get it fixed.

Thanks,
Will

OK, here’s a test case. 2 variables, 2 callbacks. one callback sends back a zero value with a timestamp, the other sends a zero value without a timestamp. First one fails, second one works, every time.

Callback_test.zip (2.7 KB)

any progress on this?
thanks,
Will

Hi there, I did not have a chance to see your code last week, I hope to give it a try this week to give you additional troubleshoot hints.

All the best

thanks. I look forward to hearing what you find.

unless you find something broken in the code, I’m done troubleshooting. it fails every single time with a timestamp, and works every single time without one.

Will