OK I think I've figured out a way to put everything in one operation by basically having:

sigmoid(x)=max(1/2, 1/(1+e^(-x)) + min(1/2, e^x/(1+e^x)) - 1/2

The output is now

[5.960464477539063e-08, 5.960464477539063e-08, 5.960464477539063e-08, 5.960464477539063e-08, 5.960464477539063e-08]
[-5.960464477539063e-08, -5.960464477539063e-08, -5.960464477539063e-08, -5.960464477539063e-08, -5.960464477539063e-08]

for the example in #1090

I've found it to be slightly slower than the original logistic function but not by much (600ms/800ms). @neubig if you can test in your environment maybe and let me know the result that would help.

TODO:

• confirm speed
• Estimate cost (I have no idea how this works, maybe @redpony can help?)