Mysql – Connecting to MYSQL on an EC2 instance

MySQL

I'm trying to use Zapier to connect to our MySQL database in order to add results from forms (hosting with WordPress) directly into our internal database.

The database is working properly and so are the forms but when I try to establish a connection to the database with Zapier I just get a timeout error.

I've spoken to Zapier about it and they've said that something is blocking them from being able to talk to our database and that I just need to ssh into the box that has the mysql client installed. I know I'm probably being very dumb but I don't understand what they mean. I asked them to clarify and they suggested I tried mysql --connect_timeout 5 -u zapper -p xxxxxxxxxxx I tried to connect to what I thought was the instance using ssh -i ~/.ssh/aws.pem -p 1903 ubuntu@12.34.5.678 (I can do that without a problem) and then tried mysql --connect_timeout 5 -u zapper -p xxxxxxxxxxx

That gave me an error saying:

Warning: Using unique option prefix pass instead of password is deprecated and will be removed in a future release. Please use the full name instead.

Enter password:

I tried my admin password, the MySQL password, the password for the zapper user as well as the sftp password but none of them work. It just says:

ERROR 1045 (28000): Access denied for user 'zapier'@'localhost' (using password: YES)

I'd be grateful for any help on this please.

EDIT:

I think I might have been able to connect. I tried changing -p to password and the got this:

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Usage: mysql [OPTIONS] [database]
-?, –help Display this help and exit.
-I, –help Synonym for -?
–auto-rehash Enable automatic rehashing. One doesn't need to use
'rehash' to get table and field completion, but startup
and reconnecting may take a longer time. Disable with
–disable-auto-rehash.
(Defaults to on; use –skip-auto-rehash to disable.)
-A, –no-auto-rehash
No automatic rehashing. One has to use 'rehash' to get
table and field completion. This gives a quicker start of
mysql and disables rehashing on reconnect.
–auto-vertical-output
Automatically switch to vertical output mode if the
result is wider than the terminal width.
-B, –batch Don't use history file. Disable interactive behavior.
(Enables –silent.)
–character-sets-dir=name
Directory for character set files.
–column-type-info Display column type information.
-c, –comments Preserve comments. Send comments to the server. The
default is –skip-comments (discard comments), enable
with –comments.
-C, –compress Use compression in server/client protocol.
-#, –debug[=#] This is a non-debug version. Catch this and exit.
–debug-check Check memory and open file usage at exit.
-T, –debug-info Print some debug info at exit.
-D, –database=name Database to use.
–default-character-set=name
Set the default character set.
–delimiter=name Delimiter to be used.
–enable-cleartext-plugin
Enable/disable the clear text authentication plugin.
-e, –execute=name Execute command and quit. (Disables –force and history
file.)
-E, –vertical Print the output of a query (rows) vertically.
-f, –force Continue even if we get an SQL error.
-G, –named-commands
Enable named commands. Named commands mean this program's
internal commands; see mysql> help . When enabled, the
named commands can be used from any line of the query,
otherwise only from the first line, before an enter.
Disable with –disable-named-commands. This option is
disabled by default.
-i, –ignore-spaces Ignore space after function names.
–init-command=name SQL Command to execute when connecting to MySQL server.
Will automatically be re-executed when reconnecting.
–local-infile Enable/disable LOAD DATA LOCAL INFILE.
-b, –no-beep Turn off beep on error.
-h, –host=name Connect to host.
-H, –html Produce HTML output.
-X, –xml Produce XML output.
–line-numbers Write line numbers for errors.
(Defaults to on; use –skip-line-numbers to disable.)
-L, –skip-line-numbers
Don't write line number for errors.
-n, –unbuffered Flush buffer after each query.
–column-names Write column names in results.
(Defaults to on; use –skip-column-names to disable.)
-N, –skip-column-names
Don't write column names in results.
–sigint-ignore Ignore SIGINT (CTRL-C).
-o, –one-database Ignore statements except those that occur while the
default database is the one named at the command line.
–pager[=name] Pager to use to display results. If you don't supply an
option, the default pager is taken from your ENV variable
PAGER. Valid pagers are less, more, cat [> filename],
etc. See interactive help (\h) also. This option does not
work in batch mode. Disable with –disable-pager. This
option is disabled by default.
-p, –password[=name]
Password to use when connecting to server. If password is
not given it's asked from the tty.
-P, –port=# Port number to use for connection or 0 for default to, in
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/services, built-in default (3306).
–prompt=name Set the mysql prompt to this value.
–protocol=name The protocol to use for connection (tcp, socket, pipe,
memory).
-q, –quick Don't cache result, print it row by row. This may slow
down the server if the output is suspended. Doesn't use
history file.
-r, –raw Write fields without conversion. Used with –batch.
–reconnect Reconnect if the connection is lost. Disable with
–disable-reconnect. This option is enabled by default.
(Defaults to on; use –skip-reconnect to disable.)
-s, –silent Be more silent. Print results with a tab as separator,
each row on new line.
-S, –socket=name The socket file to use for connection.
–ssl Enable SSL for connection (automatically enabled with
other flags).
–ssl-ca=name CA file in PEM format (check OpenSSL docs, implies
–ssl).
–ssl-capath=name CA directory (check OpenSSL docs, implies –ssl).
–ssl-cert=name X509 cert in PEM format (implies –ssl).
–ssl-cipher=name SSL cipher to use (implies –ssl).
–ssl-key=name X509 key in PEM format (implies –ssl).
–ssl-verify-server-cert
Verify server's "Common Name" in its cert against
hostname used when connecting. This option is disabled by
default.
-t, –table Output in table format.
–tee=name Append everything into outfile. See interactive help (\h)
also. Does not work in batch mode. Disable with
–disable-tee. This option is disabled by default.
-u, –user=name User for login if not current user.
-U, –safe-updates Only allow UPDATE and DELETE that uses keys.
-U, –i-am-a-dummy Synonym for option –safe-updates, -U.
-v, –verbose Write more. (-v -v -v gives the table output format).
-V, –version Output version information and exit.
-w, –wait Wait and retry if connection is down.
–connect-timeout=# Number of seconds before connection timeout.
–max-allowed-packet=#
The maximum packet length to send to or receive from
server.
–net-buffer-length=#
The buffer size for TCP/IP and socket communication.
–select-limit=# Automatic limit for SELECT when using –safe-updates.
–max-join-size=# Automatic limit for rows in a join when using
–safe-updates.
–secure-auth Refuse client connecting to server if it uses old
(pre-4.1.1) protocol.
–server-arg=name Send embedded server this as a parameter.
–show-warnings Show warnings after every statement.
–plugin-dir=name Directory for client-side plugins.
–default-auth=name Default authentication client-side plugin to use.

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf
The following groups are read: mysql client
The following options may be given as the first argument:
–print-defaults Print the program argument list and exit.
–no-defaults Don't read default options from any option file.
–defaults-file=# Only read default options from the given file #.
–defaults-extra-file=# Read this file after the global files are read.

Variables (–variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)

auto-rehash TRUE
auto-vertical-output FALSE
character-sets-dir (No default value)
column-type-info FALSE
comments FALSE
compress FALSE
debug-check FALSE
debug-info FALSE
database mycroft
default-character-set auto
delimiter ;
enable-cleartext-plugin FALSE
vertical FALSE
force FALSE
named-commands FALSE
ignore-spaces FALSE
init-command (No default value)
local-infile FALSE
no-beep FALSE
host (No default value)
html FALSE
xml FALSE
line-numbers TRUE
unbuffered FALSE
column-names TRUE
sigint-ignore FALSE
port 3306
prompt mysql>
quick FALSE
raw FALSE
reconnect TRUE
socket /var/run/mysqld/mysqld.sock
ssl FALSE
ssl-ca (No default value)
ssl-capath (No default value)
ssl-cert (No default value)
ssl-cipher (No default value)
ssl-key (No default value)
ssl-verify-server-cert FALSE
table FALSE
user zapier_connect
safe-updates FALSE
i-am-a-dummy FALSE
connect-timeout 5
max-allowed-packet 16777216
net-buffer-length 16384
select-limit 1000
max-join-size 1000000
secure-auth FALSE
show-warnings FALSE
plugin-dir (No default value)
default-auth (No default value)

Does that mean I've managed to connect? If it does then Zapier said that means it's something to do with the instance or AWS security as to why it won't connect. I've never used AWS before so don't know exactly what this means.

Best Answer

Since Zapier is coming in from outside your instance, a possible cause of your problem is the security group for your instance. Most ports on EC2 instances (except the SSH port) are normally blocked by default unless you explicitly open them.

So you would need to open the mysql port 3306 on your instance up to the ip address Zapier is coming in from. (preferably you would not open it up to everyone in the world.)

See Using Amazon RDS/EC2 on the Zapier site, starting from the section where it says Can't Connect to My Database. They continue on to provide specific instructions for the security group settings.

You will also probably need to grant permission on the database using the mysql GRANT command to the user coming in from the Zapier ip address.

Something like this should work :

GRANT ALL PRIVILEGES
ON mydb.*
TO 'user'@'54.86.9.50'
IDENTIFIED BY 'newpassword';

Or you can allow that user to come in from any ip adress like this:

GRANT ALL PRIVILEGES
ON mydb.*
TO 'user'@'%'
IDENTIFIED BY 'newpassword';

(Incoming requests would still be limited to Zapier ip addresses due to the security group settings.)

Give it a Try !!!

Mysql – Why are simple SELECTs on InnoDB 100x slower than on MyISAM

YOUR QUERY

SELECT post.postid, post.attach FROM newbb_innopost AS post WHERE post.threadid = 51506;

At first glance, that query should only touches 1.1597% (62510 out of 5390146) of the table. It should be fast given the key distribution of threadid 51506.

REALITY CHECK

No matter which version of MySQL (Oracle, Percona, MariaDB) you use, none of them can fight to one enemy they all have in common : The InnoDB Architecture.

InnoDB Architecture

CLUSTERED INDEX

Please keep in mind that the each threadid entry has a primary key attached. This means that when you read from the index, it must do a primary key lookup within the ClusteredIndex (internally named gen_clust_index). In the ClusteredIndex, each InnoDB page contains both data and PRIMARY KEY index info. See my post Best of MyISAM and InnoDB for more info.

REDUNDANT INDEXES

You have a lot of clutter in the table because some indexes have the same leading columns. MySQL and InnoDB has to navigate through the index clutter to get to needed BTREE nodes. You should reduced that clutter by running the following:

ALTER TABLE newbb_innopost
    DROP INDEX threadid,
    DROP INDEX threadid_2,
    DROP INDEX threadid_visible_dateline,
    ADD INDEX threadid_visible_dateline_index (`threadid`,`visible`,`dateline`,`userid`)
;

Why strip down these indexes ?

The first three indexes start with threadid
threadid_2 and threadid_visible_dateline start with the same three columns
threadid_visible_dateline does not need postid since it's the PRIMARY KEY and it's embedded

BUFFER CACHING

The InnoDB Buffer Pool caches data and index pages. MyISAM only caches index pages.

Just in this area alone, MyISAM does not waste time caching data. That's because it's not designed to cache data. InnoDB caches every data page and index page (and its grandmother) it touches. If your InnoDB Buffer Pool is too small, you could be caching pages, invalidating pages, and removing pages all in one query.

TABLE LAYOUT

You could shave of some space from the row by considering importthreadid and importpostid. You have them as BIGINTs. They take up 16 bytes in the ClusteredIndex per row.

You should run this

SELECT importthreadid,importpostid FROM newbb_innopost PROCEDURE ANALYSE();

This will recommend what data types these columns should be for the given dataset.

CONCLUSION

MyISAM has a lot less to contend with than InnoDB, especially in the area of caching.

While you revealed the amount of RAM (32GB) and the version of MySQL (Server version: 10.0.12-MariaDB-1~trusty-wsrep-log mariadb.org binary distribution, wsrep_25.10.r4002), there are still other pieces to this puzzle you have not revealed

The InnoDB settings
The Number of Cores
Other settings from my.cnf

If you can add these things to the question, I can further elaborate.

UPDATE 2014-08-28 11:27 EDT

You should increase threading

innodb_read_io_threads = 64
innodb_write_io_threads = 16
innodb_log_buffer_size = 256M

I would consider disabling the query cache (See my recent post Why query_cache_type is disabled by default start from MySQL 5.6?)

query_cache_size = 0

I would preserve the Buffer Pool

innodb_buffer_pool_dump_at_shutdown=1
innodb_buffer_pool_load_at_startup=1

Increase purge threads (if you do DML on multiple tables)

innodb_purge_threads = 4